Validation-Based Hyperparameter Optimization of a Hybrid IndoBERT-BiGRU Model for Indonesian Sentiment Analysis

Handoko; Elly; Rivaldi Lubis

doi:10.21063/jtif.2026.V14.2.969

Authors

Handoko Universitas Mikroskill
Elly Universitas Mikroskill
Rivaldi Lubis Universitas Mikroskill

DOI:

https://doi.org/10.21063/jtif.2026.V14.2.969

Keywords:

BiGRU, Domain-aware Model, Hybrid Sentiment Analysis, Hyperparameter Optimization, Pseudo-labeling

Abstract

Public sentiment across digital platforms spans multiple domains, each with different linguistic characteristics. This study investigated the hyperparameter configurations selected for a hybrid IndoBERT-BiGRU model across three Indonesian-language datasets and an aggregated dataset. Sentiment pseudo-labels were generated using the Indonesian Sentiment Lexicon (InSet), while Optuna with a tree-structured Parzen estimator (TPE) optimized BiGRU hidden size, number of layers, and dropout rate for the four training settings. Validation Macro F1 was used as the optimization objective. The Global and MyPertamina models selected 384 hidden units, two BiGRU layers, and a dropout rate of 0.2465; the Shin Tae-yong model selected 256 hidden units, one layer, and a dropout rate of 0.2301; and the One For All model selected 128 hidden units, one layer, and a dropout rate of 0.3543. Their validation Macro F1-scores were 81.25%, 82.16%, 61.40%, and 65.71%. Agreement between InSet-generated pseudo-labels and Human_Sentiment labels across 14,526 records, measured using Cohen’s kappa, yielded a value of 0.2511. On the held-out test sets, evaluated against InSet-derived pseudo-labels, the models achieved Macro F1-scores of 49.19%, 46.86%, 22.71%, and 30.00% for Global, MyPertamina, Shin Tae-yong, and One For All, respectively. Per-class evaluation showed consistently low F1-scores for the Neutral class. These findings demonstrate differences in the validation-selected configurations but do not establish that domain-specific tuning is superior or that dataset size determines model complexity. The results should be interpreted in light of the limited agreement between InSet and Human_Sentiment labels and the very small One For All test set, which contains only eight records.

References

[1] M. He, C. Ma, and R. Wang, “A Data-Driven Approach for University Public Opinion Analysis and Its Applications,” Applied Sciences, vol. 12, no. 18, 2022, doi: https://doi.org/10.3390/app12189136.

[2] T. Anderson, S. Sarkar, and R. Kelley, “Analyzing Public Sentiment on Sustainability: A Comprehensive Review and Application of Sentiment Analysis Techniques,” Natural Language Processing Journal, vol. 8, p. 100097, 2024, doi: https://doi.org/10.1016/j.nlp.2024.100097.

[3] S. J. Pipin and H. Kurniawan, “Analisis Sentimen Kebijakan MBKM Berdasarkan Opini Masyarakat di Twitter Menggunakan LSTM,” Jurnal SIFO Mikroskil, vol. 23, no. 2, pp. 197–208, Oct. 2022, doi: https://doi.org/10.55601/jsm.v23i2.900.

[4] M. Wankhade, A. C. S. Rao, and C. Kulkarni, “A Survey on Sentiment Analysis Methods, Applications, and Challenges,” Artif. Intell. Rev., vol. 55, no. 7, pp. 5731–5780, 2022, doi: https://doi.org/10.1007/s10462-022-10144-1.

[5] J. Cui, Z. Wang, S.-B. Ho, and E. Cambria, “Survey on Sentiment Analysis: Evolution of Research Methods and Topics,” Artif. Intell. Rev., vol. 56, no. 8, pp. 8469–8510, 2023, doi: https://doi.org/10.1007/s10462-022-10386-z.

[6] M. El Barachi, M. AlKhatib, S. Mathew, and F. Oroumchian, “A Novel Sentiment Analysis Framework for Monitoring the Evolving Public Opinion in Real-Time: Case Study on Climate Change,” J. Clean. Prod., vol. 312, p. 127820, 2021, doi: https://doi.org/10.1016/j.jclepro.2021.127820.

[7] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding,” in North American Chapter of the Association for Computational Linguistics, vol. 1, pp. 4171–4186, 2019, doi: 10.18653/v1/N19-1423.

[8] F. Koto, A. Rahimi, J. H. Lau, and T. Baldwin, “IndoLEM and IndoBERT: A Benchmark Dataset and Pre-trained Language Model for Indonesian NLP,” in Proceedings of the 28th International Conference on Computational Linguistics, D. Scott, N. Bel, and C. Zong, Eds., Barcelona, Spain (Online): International Committee on Computational Linguistics, Dec. 2020, pp. 757–770. doi: https://doi.org/10.18653/v1/2020.coling-main.66.

[9] J. Blitzer, M. Dredze, and F. Pereira, “Biographies, Bollywood, Boom-boxes and Blenders: Domain Adaptation for Sentiment Classification,” in Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, A. Zaenen and A. den Bosch, Eds., Prague, Czech Republic: Association for Computational Linguistics, Jun. 2007, pp. 440–447. [Online]. Available: https://aclanthology.org/P07-1056/

[10] Q. Zhu, X. Jiang, and R. Ye, “Sentiment Analysis of Review Text Based on BiGRU-Attention and Hybrid CNN,” IEEE Access, vol. 9, pp. 149077–149088, 2021, doi: https://doi.org/10.1109/ACCESS.2021.3118537.

[11] G. Kaur and A. Sharma, “HAS: Hybrid Analysis of Sentiments for the Perspective of Customer Review Summarization,” J. Ambient Intell. Humaniz. Comput., vol. 14, no. 9, pp. 11971–11984, 2023, doi: https://doi.org/10.1007/s12652-022-03748-6.

[12] H. Ahmadian, T. F. Abidin, H. Riza, and K. Muchtar, “Hybrid Models for Emotion Classification and Sentiment Analysis in Indonesian Language,” Applied Computational Intelligence and Soft Computing, vol. 2024, pp. 1–17, 2024, doi: https://doi.org/10.1155/2024/2826773.

[13] W. Zhang, X. Li, Y. Deng, L. Bing, and W. Lam, “A Survey on Aspect-Based Sentiment Analysis: Tasks, Methods, and Challenges,” IEEE Transactions on Knowledge & Data Engineering , vol. 35, no. 11, pp. 11019–11038, Nov. 2023, doi: https://doi.org/10.1109/TKDE.2022.3230975.

[14] Y. Fu and Y. Liu, “Contrastive transformer based domain adaptation for multi-source cross-domain sentiment classification,” Knowledge-Based Systems, vol. 245, Art. no. 108649, 2022, doi: 10.1016/j.knosys.2022.108649.

[15] Y. Zhu, Y. Qiu, Q. Wu, F. L. Wang, and Y. Rao, “Topic Driven Adaptive Network for cross-domain sentiment classification,” Information Processing & Management, vol. 60, no. 2, Art. no. 103230, 2023, doi: 10.1016/j.ipm.2022.103230.

[16] S. Z. Pranida and A. Kurniawardhani, “Sentiment Analysis of Expedition Customer Satisfaction using BiGRU and BiLSTM,” Indonesian Journal of Artificial Intelligence and Data Mining, vol. 5, no. 1, pp. 44–53, Jun. 2022, doi: https://doi.org/10.24014/ijaidm.v5i1.17361.

[17] T. Akiba, S. Sano, T. Yanase, T. Ohta, and M. Koyama, “Optuna: A Next-generation Hyperparameter Optimization Framework,” in Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, in KDD ’19. New York, NY, USA: Association for Computing Machinery, 2019, pp. 2623–2631. doi: https://doi.org/10.1145/3292500.3330701.

[18] X. Liu and C. Wang, “An Empirical Study on Hyperparameter Optimization for Fine-Tuning Pre-trained Language Models,” in Proc. 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), 2021, pp. 2286–2300, doi: 10.18653/v1/2021.acl-long.178.

[19] D. Salinas, H. Shen, and V. Perrone, “A Quantile-based Approach for Hyperparameter Transfer Learning,” in Proc. 37th International Conference on Machine Learning, Proc. Mach. Learn. Res., vol. 119, 2020, pp. 8438–8448, doi: https://doi.org/10.48550/arXiv.1909.13595

[20] M. R. Grootendorst, “BERTopic: Neural Topic Modeling With a Class-Based Tf-Idf Procedure,” ArXiv, vol. abs/2203.05794, 2022, doi: https://doi.org/10.48550/arXiv.2203.05794.

[21] M. Ali et al., “Investigating Mental Wellbeing Self-Care in Higher Education Using BERTopic Modeling,” Discover Mental Health, vol. 5, no. 1, p. 204, Nov. 2025, doi: https://doi.org/10.1007/s44192-025-00323-1.

[22] A. Rejeb, K. Rejeb, S. Simske, and E. Süle, “Industry 5.0 Research: An Approach Using Co-word Analysis and BERTopic Modeling,” Discover Sustainability, vol. 6, no. 1, p. 402, 2025, doi: https://doi.org/10.1007/s43621-025-01252-3.

[23] sagitasantia, “Sentiment Analysis MyPertamina Application,” Kaggle. [Online]. Available: https://www.kaggle.com/datasets/sagitasantia/sentiment-analisis-mypertamina-application. Accessed: Feb. 18, 2026.

[24] wildanabid19, “Analisis Sentiment STY Raw,” Kaggle. [Online]. Available: https://www.kaggle.com/datasets/wildanabid19/analisis-sentiment-sty-raw. Accessed: Feb. 18, 2026.

[25] ldausl, “Merah Putih: One for All Sentiment,” Kaggle. [Online]. Available: https://www.kaggle.com/datasets/ldausl/merah-putih-one-for-all-sentiment. Accessed: Feb. 18, 2026.

[26] N. Garg and K. Sharma, “Text Pre-processing of Multilingual for Sentiment Analysis Based on Social Network Data,” International Journal of Electrical and Computer Engineering (IJECE), vol. 12, no. 1, pp. 776–784, 2022, doi: https://doi.org/10.11591/ijece.v12i1.pp776-784.

[27] L. Hickman, S. Thapa, L. Tay, M. Cao, and P. Srinivasan, “Text Preprocessing for Text Mining in Organizational Research: Review and Recommendations,” Organ. Res. Methods, vol. 25, no. 1, pp. 114–146, 2022, doi: https://doi.org/10.1177/1094428120971683.

[28] F. Koto and G. Y. Rahmaningtyas, “Inset Lexicon: Evaluation of a Word List for Indonesian Sentiment Analysis in Microblogs,” in 2017 International Conference on Asian Language Processing (IALP), 2017, pp. 391–394. doi: https://doi.org/10.1109/IALP.2017.8300625.

[29] R. Wijayanti and A. Arisal, “Automatic Indonesian Sentiment Lexicon Curation with Sentiment Valence Tuning for Social Media Sentiment Analysis,” ACM Trans. Asian Low-Resour. Lang. Inf. Process., vol. 20, no. 1, Mar. 2021, doi: https://doi.org/10.1145/3425632.

[30] Y. Zhang, M. Jiang, Y. Meng, Y. Zhang, and J. Han, “PIEClass: Weakly-Supervised Text Classification with Prompting and Noise-Robust Iterative Ensemble Training,” in Proc. Conference on Empirical Methods in Natural Language Processing (EMNLP), 2023, pp. 12655–12670, doi: 10.18653/v1/2023.emnlp-main.780.

[31] E. Dave and A. Chowanda, “IPerFEX-2023: Indonesian Personal Financial Entity Extraction Using indoBERT-BiGRU-Crf Model,” J. Big Data, vol. 11, no. 1, p. 139, 2024, doi: https://doi.org/10.1186/s40537-024-00987-6.

[32] J. Hestness et al., “Deep Learning Scaling is Predictable, Empirically,” ArXiv, vol. abs/1712.00409, 2017, doi: https://doi.org/10.48550/arXiv.1712.00409.