A cryptographic test of randomness, entropy, and brute force attack on the modified playfair algorithm with the novel dynamic matrix

Abstract

One critical concern in designing a cryptographic algorithm is its randomness. The randomness test examines the quality of random numbers generated by cryptographic algorithms. In this paper, the National Institute of Standards and Technology (NIST) Statistical Test Suite for Random and Pseudorandom Number Generators for Cryptographic Applications was instrumental in assessing the security of the modified Playfair algorithm with the novel Multidimensional Element-in-Grid Sequencer (MEGS). Moreover, this study determines the level of security of the MEGS-based Playfair algorithm against brute force attacks. Simulation results revealed that the modified cryptographic algorithm could execute the encryption process with randomly generated ciphertext despite having repetitive letters in the plaintext, as evident in the frequency (monobit) test, frequency within a block test, and runs test with p-values > 0.01. Further, the MEGS-based Playfair algorithm was highly resistant against attacks based on the results of the entropy test and the brute force attack analysis. Since the character keyspace of the modified algorithm is 256, attempts to break the ciphertext would cost a large amount of financial and computational resources. The use of a 16x16 dynamic matrix with matrix rotation, matrix shifting, matrix rolling, and crossover operations enhanced the performance of the Playfair algorithm, improving its resilience to attacks.