Klasifikasi Teks Menggunakan Pemodelan FineTuning IndoBERT dan Deep Learning

• Alda Zevana Putri Widodo
• 14210192

---

ABSTRAK

ABSTRAK

Nama               : Alda Zevana Putri Widodo

NIM                  : 14210192

Program Studi : Ilmu Komputer

Fakultas           : Teknologi Informasi

Jenjang             : Strata Dua (S2)

Konsentrasi       : Data Mining

Judul                 : Klasifikasi Teks Menggunakan Pemodelan FineTuning IndoBERT dan Deep Learning

Perkembangan teknologi dalam fasilitas pengiriman barang semakin meningkat dari tahun ke tahun sejalan dengan meningkatnya perdagangan online yang membutuhkan jasa pengiriman untuk memenuhi proses transaksi antara penjual dan pembeli. Sejak tahun 2000 penghargaan top brand sering melakukan analisa survei resmi dalam memberikan perbandingan barang atau jasa, salah satunya adalah layanan pengiriman. Namun peringkat survei tersebut kurang akurat dikarenakan masyarakat pengguna jasa pengiriman dan perusahaan jasa belum mengetahui detail faktor keberhasilan dan kelemahan dari jasa mereka. Penelitian ini bertujuan menjadi bahan evaluasi kualitas pelayanan jasa pengiriman terhadap kepuasan konsumen pengguna layanan J&T Express berdasarkan 4 faktor yaitu harga, waktu, pengembalian, dan lainnya. Data penelitian yang digunakan sebanyak 2525 data komentar pengguna akun Twitter terhadap jasa pengiriman tersebut dari rentang waktu 1 Januari 2021 – 31 Maret 2023. Penelitian ini mengusulkan fine-tuning IndoBert base model dari IndoNLU dengan arsitektur algoritma Deep Neural Network yaitu CNN dan Bi-LSTM untuk mengetahui performansi input Bert Embedding dengan beberapa arsitektur model algoritma deep neural network. Akurasi 83% diperoleh CNN dengan parameter 1x10-6 dengan teknik preprocessing terlebih dahulu dinilai lebih tinggi dibandingan Bi-LSTM.

KATA KUNCI

Twitter,INDOBERT,Pengiriman

---

DAFTAR PUSTAKA

DAFTAR PUSTAKA

[1] I. R. M. Association, Research Anthology on Social Media Advertising and Building Consumer Relationships. IGI Global, 2022.

[2] S. Smith, Social Media Marketing for Business 2020. Samuel Smith, 2021.

[3] N. F. Rahmah, “The Influence of Service Quality on Consumer satisfaction Delivery Services at JNT Express in Garut,” bircu, vol. 5, no. 3, p. 13, 2022, doi: https://doi.org/10.33258/birci.v5i3.6628.

[4] F. Dwita, “THE EFFECT OF TOXIC LEADERSHIP AND JOB STRESS ON TURNOVER INTENTION IN LOGISTIC COURIER BEKASI CITY,” Airlangga J. Innov. Manag., vol. 3, no. 2, p. 10, 2022, doi: 10.20473/ajim.v3i1.39982.

[5] A. S. A. Ali, “CNN-Bi LSTM Neural Network for Simulating Groundwater Level,” Comput. Res. Prog. Appl. Sci. Eng., p. 7, 2022, doi: https://doi.org/10.52547/crpase.8.1.2748.

[6] M. Pota, “An Effective BERT-Based Pipeline for Twitter Sentiment Analysis: A Case Study in Italian,” Sensors, p. 21, 2020, doi: https://doi.org/10.3390/s21010133.

[7] L. P. M. Aliyah Kurniasih, “On the Role of Text Preprocessing in BERT Embedding-based DNNs for Classifying Informal Texts,” Aliyah Kurniasih, vol. 13, p. 8, 2022.

[8] E. Bakana, Service Delivery and Customer Satisfaction. The Case of Burayu Town Municipality, Ethiopia. GRIN Verlag, 2020.

[9] B. Vincent, Twitter Marketing. Revival Waves of Glory, 2021.

[10] D. NEUPANE, “Bearing Fault Detection and Diagnosis Using Case Western Reserve University Dataset With Deep Learning Approaches: A Review,” IEEE Access, vol. 8, no. 5, p. 24, 2020, doi: 10.1109/ACCESS.2020.2990528. 42 Program Studi Ilmu Komputer (S2) Universitas Nusa Mandiri

[11] N. M. Ali, “Sentiment Analysis for Movies Reviews Dataset Using Deep Learning Models,” SSRN, vol. 9, no. 2, p. 9, 2019, doi: 10.5121/ijdkp.2019.9302.

[12] W.-M. Lee, Python Machine Learning. Indianapolis: John Wiley & Sons, Inc, 2019.

[13] Davidson-Pilon, “lifelines: survival analysis in Python,” J. Open Source Softw., p. 3, 2019, doi: https://doi.org/10.21105/joss.01317.

[14] bhishek A. Chaudhri, “Implementation Paper on Analyzing COVID-19 Vaccines on Twitter Dataset Using Tweepy and TextBlob,” Ann. R.S.C.B, vol. 25, no. 3, pp. 8393–8396, 2021, doi: 1583-6258.

[15] J. Chaudhary, “Twitter Sentiment Analysis using Tweepy,” Int. Res. J. Eng. Technol., vol. 08, no. 04, p. 5, 2021, doi: 2395-0056.

[16] N. Jiang, “Impact of Code Language Models on Automated Program Repair,” IEEE/ACM 45th Int. Conf. Softw. Eng., p. 13, 2023, doi: https://doi.org/10.48550/arXiv.2302.05020.

[17] M. Chen, “Evaluating Large Language Models Trained on Code,” IEEE, vol. 2, no. 3, p. 35, 2021, doi: https://doi.org/10.48550/arXiv.2107.03374.

[18] A. George, Python Text Mining. BPB Publications, 2022.

[19] J. Yang, “Brief introduction of medical database and data mining technology in big data era,” J. Evid. Based. Med., vol. 13, no. 1, pp. 57–69, 2020, doi: https://doi.org/10.1111/jebm.12373.

[20] R. Jafari, Hands-On Data Preprocessing in Python. Packt Publishing, 2022.

[21] S. TANG, “Data Preprocessing Techniques in Convolutional Neural Network Based on Fault Diagnosis Towards Rotating Machinery,” IEEE Access, vol. 8, p. 10, 2020, doi: 10.1109/ACCESS.2020.3012182.

[22] S. Sarkar, Ed., Intelligent Multi-Modal Data Processing, First Edit. Chennai: 43 Program Studi Ilmu Komputer (S2) Universitas Nusa Mandiri John Wiley & Sons, Inc, 2021.

[23] D. A. Leon, Data Processing with Optimus?: Supercharge Big Data Preparation Tasks for Analytics and Machine Learning with Optimus Using Dask and PySpark. Birmingham: Packt Publishing, 2021.

[24] R. Jafari, Hands-On Data Preprocessing in Python?: Learn how to Effectively Prepare Data for Successful Data Analytics. Birmingham: Packt Publishing, 2022.

[25] A. S. Akshay Kulkarni, Natural Language Processing Recipes. Apress, 2019.

[26] S. Alaparthi, “Bidirectional Encoder Representations from Transformers (BERT): A sentiment analysis odyssey,” Comput. Lang., p. 15, 2020, doi: https://doi.org/10.48550/arXiv.2007.01127.

[27] C. Sun, How to Fine-Tune BERT for Text Classification? Shanghai: Springer Nature Switzerland AG, 2019. doi: https://doi.org/10.1007/978-3-030-32381- 3_16.

[28] A. Aghajanyan, “BETTER FINE-TUNING BY REDUCING REPRESENTATIONAL COLLAPSE,” Comput. Lang., p. 12, 2020, doi: https://doi.org/10.48550/arXiv.2008.03156.

[29] A. McMillan-Major, “Reusable Templates and Guides For Documenting Datasets and Models for Natural Language Processing and Generation?: A Case Study of the HuggingFace and GEM Data and Model Cards,” Comput. Lang., p. 15, 2021, doi: https://doi.org/10.48550/arXiv.2108.07374.

[30] T. Nguyen, “Deep neural network with high-order neuron for the prediction of foamed concrete strength,” Comput. Civ. Infrastruct. Eng., vol. 34, no. 4, pp. 316–332, 2019, doi: https://doi.org/10.1111/mice.12422.

[31] W. Lu, “A CNN-BiLSTM-AM method for stock price prediction,” Neural Comput. Appl., vol. 33, no. 4, pp. 4741–4753, 2020, doi: https://doi.org/10.1007/s00521-020-05532-z. 44 Program Studi Ilmu Komputer (S2) Universitas Nusa Mandiri

[32] F. Theis, Artificial Neural Networks and Machine Learning – ICANN 2019: Text and Time Series. Springer International Publishing, 2019.

[33] M. Tan, “EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks,” Int. Conf. Mach. Learn. Proc. Mach. Learn. Res., vol. 97, pp. 6105– 6114, 2019.

[34] A. Ghosh, “Fundamental Concepts of Convolutional Neural Network,” Intell. Syst. Ref. Libr., vol. 172, 2020, doi: https://doi.org/10.1007/978-3-030-32644- 9_36.

[35] S. R. Dubey, “diffGrad: An Optimization Method for Convolutional Neural Networks,” IEEE Trans. Neural Networks Learn. Syst., vol. 31, no. 11, pp. 4500–4511, 2020, doi: 10.1109/TNNLS.2019.2955777.

[36] F. Millstein, Convolutional Neural Networks In Python?: Beginner’s Guide To Convolutional Neural Networks In Python. Chicago: Frank Millstein, 2020.

[37] I. Kouretas, “Simplified Hardware Implementation of the Softmax Activation Function,” IEEE, 2019, doi: https://doi.org/10.1109/MOCAST.2019.8741677.

[38] H. Pratiwi, “Sigmoid Activation Function in Selecting the Best Model of Artificial Neural Networks,” J. Phys. Conf. Ser., vol. 1471, p. 8, 2020, doi: 10.1088/1742-6596/1471/1/012010.

[39] S. K. Roy, “LiSHT: Non-parametric Linearly Scaled Hyperbolic Tangent Activation Function for Neural Networks,” Comput. Vis. Image Process., vol. 1776, pp. 462–476, 2023, doi: https://doi.org/10.1007/978-3-031-31407-0_35.

[40] O. Calin, Deep Learning Architectures: A Mathematical Approach. Jerman: Springer International Publishing, 2020.

[41] A. Géron, Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow. Amerika Serikat: O’Reilly Media, 2022.

[42] Y. YU, “RMAF: Relu-Memristor-Like Activation Function for Deep Learning,” 45 Program Studi Ilmu Komputer (S2) Universitas Nusa Mandiri IEEE Access, vol. 8, pp. 72727–72741, 2020, doi: 10.1109/ACCESS.2020.2987829.

[43] A. Labach, “Survey of Dropout Methods for Deep Neural Networks,” Neural Evol. Comput., vol. 2, p. 13, 2019, doi: https://doi.org/10.48550/arXiv.1904.13310.

[44] B. A. Skourt, “Mixed-pooling-dropout for convolutional neural network regularization,” J. King Saud Univ. – Comput. Inf. Sci., vol. 34, no. 8, p. 6, 2022, doi: https://doi.org/10.1016/j.jksuci.2021.05.001.

[45] X. Liang, “R-Drop: Regularized Dropout for Neural Networks,” Neural Inf. Process. Syst., vol. 5, p. 16, 2021.

[46] I. Kandel, “The effect of batch size on the generalizability of the convolutional neural networks on a histopathology dataset,” ScienceDirect, vol. 6, p. 4, 2020, doi: https://doi.org/10.1016/j.icte.2020.04.010.

[47] N. Aneja, “Transfer Learning using CNN for Handwritten Devanagari Character Recognition,” IEEE, 2019, doi: https://doi.org/10.1109/ICAIT47043.2019.8987286.

[48] K. Osawa, “Practical Deep Learning with Bayesian Principles,” Neural Inf. Process. Syst., vol. 12, p. 13, 2019.

[49] L. F. W. Anthony, “Carbontracker: Tracking and Predicting the Carbon Footprint of Training Deep Learning Models,” Comput. Soc., vol. 3, p. 11, 2020, doi: https://doi.org/10.48550/arXiv.2007.03051.

[50] K. K. Chandriah, “RNN / LSTM with modified Adam optimizer in deep learning approach for automobile spare parts demand forecasting,” SpringerLink, 2021.

[51] A. Glassner, Deep Learning?: A Visual Approach. Amerika Serikat: No Starch Press, 2021.

[52] K. You, “HOW DOES LEARNING RATE DECAY HELP MODERN 46 Program Studi Ilmu Komputer (S2) Universitas Nusa Mandiri NEURAL NETWORKS?,” Mach. Learn., vol. 2, p. 15, 2019, doi: https://doi.org/10.48550/arXiv.1908.01878.

[53] F. He, “Control Batch Size and Learning Rate to Generalize Well: Theoretical and Empirical Evidence,” p. 4, 2019.

[54] Henderi, “COMPARISON OF MIN-MAX NORMALIZATION AND ZSCORE NORMALIZATION IN THE K-NEAREST NEIGHBOR (KNN) ALGORITHM TO TEST THE ACCURACY OF TYPES OF BREAST CANCER,” ijiis, vol. 41, 2021.

[55] T. B. Alakus, “Comparison of deep learning approaches to predict COVID-19 infection,” Elsevier, p. 7, 2020, doi: https://doi.org/10.1016/j.chaos.2020.110120.

[56] I. Priyadarshini, “A novel LSTM–CNN–grid search-based deep neural network for sentiment analysis,” J. Supercomput. Vol., vol. 77, pp. 13911–13932, 2021, doi: https://doi.org/10.1007/s11227-021-03838-w.

[57] H. Luan, “A Review of Using Machine Learning Approaches for Precision Education,” Educ. Technol. Soc., vol. 24, no. 1, p. 17, 2021.

[58] Hyun, “A Machine-Learning Approach Using PET-Based Radiomics to Predict the Histological Subtypes of Lung Cancer,” Clin. Nucl. Med., 2019, doi: 10.1097/RLU.0000000000002810.

[59] K. Ameri, “CyBERT: Cybersecurity Claim Classification by Fine-Tuning the BERT Language Model,” MDPI, 2021, doi: https://doi.org/10.3390/jcp1040031.

[60] R. Qasim, “A Fine-Tuned BERT-Based Transfer Learning Approach for Text Classification,” Hindawi, p. 17, 2021, doi: https://doi.org/10.1155/2022/3498123.

[61] R. K. Kaliyar, “FakeBERT: Fake news detection in social media with a BERTbased deep learning approach,” Springer Link, p. 24, 2021, doi: 47 Program Studi Ilmu Komputer (S2) Universitas Nusa Mandiri https://doi.org/10.1007/s11042-020-10183-2.

[62] B. Gupta, “Integrated BERT embeddings, BiLSTM-BiGRU and 1-D CNN model for binary sentiment classification analysis of movie reviews,” Springer Link, 2022.

[63] Z. Shi, “BFCN: A Novel Classification Method of Encrypted Traffic Based on BERT and CNN,” MDPI, p. 16, 2023, doi: https://doi.org/10.3390/electronics12030516.