Docker Compose Getting Started with JAVA RocketMQ

Motivation

Just want to try Docker Compose...

Prerequisites

Install Docker Compose on Ubuntu 18.04
https://ben-bai.blogspot.com/2019/11/install-docker-compose-on-ubuntu-1804.html

JAVA JMS: RocketMQ NameServer Broker Cluster
https://ben-bai.blogspot.com/2019/10/java-jms-rocketmq-nameserver-broker.html


Simple Note

Go to Dockerfile folder,

Add the docker-compose.yml below into it
https://github.com/benbai123/JSP_Servlet_Practice/blob/master/Practice/JAVA/JMS/RocketMQ/RocketMQCluster/docker-compose.yml

then run docker-compose up

Docker Compose will build image and run container with specified config value

Testing Video
https://youtu.be/QHQEacrtU9k

Shortly,

Dockerfile define "how to create the container"

docker-compose.yml describe "how to config/run containers"


References

Get started with Docker Compose
https://docs.docker.com/compose/gettingstarted/

Compose file version 3 reference
https://docs.docker.com/compose/compose-file/

Install Docker Compose on Ubuntu 18.04

Simple Note

Just several lines of commands

# Run this command to download the current stable release of Docker Compose
# To install a different version of Compose, substitute 1.24.1 with the version of Compose you want to use.
sudo curl -L "https://github.com/docker/compose/releases/download/1.24.1/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose

# Apply executable permissions to the binary:
sudo chmod +x /usr/local/bin/docker-compose

# Test the installation.
docker-compose --version

# done

Reference

Install Docker Compose

JAVA Selenium Visual Testing

Motivation

Visual Testing is convenient, let you find any change of a web page in one line.

Prerequisites

You probably will want to view the articles in this blog below before reading this article

JAVA Selenium Browser Automation Getting Started

JAVA Selenium Commands Testing

JAVA Image Basic Operations

Java Diff Image and Highlight Differences

Simple Note

Testing file VisualTessting.java

Visual Testing Implementation VisualTestingImpl.java

After the testing file is executed, you should find some images in the folder selenium/visualTesting/123456 under your project folder.

References

Converting BufferedImage to ByteBuffer

How to make a hashcode(integer value) positive

How to take Re-entrant lock on String value?

Java Diff Image and Highlight Differences

Motivation

I want to implement visual testing with Selenium.

There is already a library romankh3/image-comparison

but it says "with the same sizes" so probably not suitable in some cases.

Simple Note

Test file ImageDiff.java

Diff implementation ImageDiffImpl.java

Assume expected image aa.png and actual image cc.png
the diff result will be output_image.png

References

How do you clone a BufferedImage

calculating distance between a point and a rectangular box

Filling a Multidimensional Array using a Stream

JAVA Image Basic Operations

Motivation

I want to implement visual testing with Selenium screenshot so need to know basic operations of Image in JAVA.

Simple Note

Testing file TestBasicOperations.java
Including read image file, get information of each pixel, draw rectangle on loaded image then save it.

Pixel class Pixel.java
for store information of each pixel

References

Java - get pixel array from image

Java 8: Replace traditional for loops with IntStreams

JAVA Selenium Commands Testing

Motivation

Create a testing file for reference.

Simple Note

Testing file: CommandsTesting.java

Basically, can do everything that can be done in JavaScript

References

Selenium WebDriver- Commands

Switch tabs using Selenium WebDriver with Java

WebDriver - wait for element using Java

How to wait until an element is present in Selenium?

JAVA Selenium Browser Automation Getting Started

Simple Note

Two versions,

use io.github.bonigarcia WebDriverManager to set up ChromeDriver
GettingStartedWithDriverManager.java

manually download chromedriver.exe and specify path
GettingStartedWithDownloadedChromeDriver.java

and this is a Maven Project so
pom.xml

Related Resources

 Download ChromeDriver here:
 https://chromedriver.chromium.org/downloads

 Check what version of Chrome do you have
 https://www.whatismybrowser.com/detect/what-version-of-chrome-do-i-have

References

Example Maven pom.xml for Java based Selenium WebDriver project for Firefox

release Selenium chromedriver.exe from memory

JAVA Dynamic Compile Class From Text

Simple Note

Test File: TestDynamicCompile.java

Note: Should use JDK as System Library, not JRE

Reference

How do I programmatically compile and instantiate a Java class?

JAVA JMS: RocketMQ NameServer Broker Cluster

Simple Note

After bind necessary ports
(see Connect to VirtualBox Guest Machine from Host OS with NAT Port Forwarding)

Creat Docker Image with Dockerfile
(see Create Dockerfile for Docker Image of RocketMQ NameServer)

Can give it a try finally...

Testing Video

https://youtu.be/mHuEyVh0eUk

The command for start broker is slightly different, add -c to use specific config and specify multiple nameServer address

e.g.

mqbroker.cmd -c ../conf/2m-2s-sync/broker-a.properties -n localhost:9487;localhost:9478 autoCreateTopicEnable=true

All files in the test project

RocketMQCluster

Create Dockerfile for Docker Image of RocketMQ NameServer

Simple Note

Please refer to this Dockerfile  and README.md

References

Docker run reference

docker build

Connect to VirtualBox Guest Machine from Host OS with NAT Port Forwarding

Motivation

I want to connect to NameServers in Docker Containers,
and the Docker installed on Ubuntu 18.04,
and the Ubuntu 18.04 running in VirtualBox on my Windows Host OS,

... so this is required.

Simple Notes

Step 1: Right click on stopped (power off) machine, click "Settings..."

Step 2: Click Network -> Advanced -> Port Forwarding

Step 3: Click the add Sign, input Host Port and Guest Port, click OK

Now the request to localhost:9487 will be forwarded to the Guest Machine

Reference

How to Forward Ports to a Virtual Machine and Use It as a Server

Install Docker on Ubuntu 18.04

Motivation

I want to try RocketMQ Cluster with Multi-NameServers/Brokers, but it seems change port of NameServer is not supported...

So want to run 2 NameServers in 2 Docker Containers and bind to different ports for testing.

Simple Note

Install Docker on Ubuntu 18.04 running in VirtualBox

# Uninstall old versions
sudo apt-get remove docker docker-engine docker.io containerd runc

# SET UP THE REPOSITORY
## Update the apt package index
sudo apt-get update

## Install packages to allow apt to use a repository over HTTPS
sudo apt-get install \
    apt-transport-https \
    ca-certificates \
    curl \
    gnupg-agent \
    software-properties-common

## Add Docker’s official GPG key
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -

## Verify that you now have the key with the fingerprint 9DC8 5822 9FC7 DD38 854A E2D8 8D81 803C 0EBF CD88, by searching for the last 8 characters of the fingerprint.
sudo apt-key fingerprint 0EBFCD88

## Use the following command to set up the stable repository
sudo add-apt-repository \
   "deb [arch=amd64] https://download.docker.com/linux/ubuntu \
   $(lsb_release -cs) \
   stable"


# INSTALL DOCKER ENGINE - COMMUNITY
## Update the apt package index.
sudo apt-get update

## Install the latest version of Docker Engine - Community and containerd, or go to the next step to install a specific version:
sudo apt-get install docker-ce docker-ce-cli containerd.io

## Verify that Docker Engine - Community is installed correctly by running the hello-world image.
sudo docker run hello-world

Reference

Get Docker Engine - Community for Ubuntu

RocketMQ not support modify nameServer port

JAVA JMS: RocketMQ Getting Started

Simple Notes

1. Download NameServer and Broker
  Go to archive.apache.org
  Download release zip file
  (I choose 4.5.2/rocketmq-all-4.5.2-bin-release.zip)

2. Extract it
  Assume I extract it to
  D:\things\Downloads\rocketmq\rocketmq-all-4.5.2-bin-release

3. Start NameServer
  Open cmd, run commands below

  cd D:\things\Downloads\rocketmq\rocketmq-all-4.5.2-bin-release\bin
  D:
  set JAVA_HOME=D:\things\programs\Java\jdk1.8.0_221
  set ROCKETMQ_HOME=D:\things\Downloads\rocketmq\rocketmq-all-4.5.2-bin-release\
  mqnamesrv.cmd -n localhost:9876

4. Start Broker
  Open cmd, run commands below

  cd D:\things\Downloads\rocketmq\rocketmq-all-4.5.2-bin-release\bin
  D:
  set JAVA_HOME=D:\things\programs\Java\jdk1.8.0_221
  set ROCKETMQ_HOME=D:\things\Downloads\rocketmq\rocketmq-all-4.5.2-bin-release\
  mqbroker.cmd -n localhost:9876 autoCreateTopicEnable=true

5. Write Consumer and Producer for test

  TestConsumer.java ; TestProducer.java ; and this is a maven project, so pom.xml
  Run TestConsumer and TestProducer, you will see something in the console.

References

Quick Start
https://rocketmq.apache.org/docs/quick-start/

Simple Message Example
https://rocketmq.apache.org/docs/simple-example/

JAVA RSA Sign Data and Verify

Simple Note

Add Sign data by Signature with Private Key and verify data by Signature with Public Key

RSAUtils.java

Also add Test Cases and Performance Testing

TestRSAUtils.java

TestRSAPerformance.java

and update the Performance Testing google sheet

JAVA RSA Performance Testing

Reference

Java Cryptography Architecture (JCA) Reference Guide
Generating and Verifying a Signature Using Generated Keys
https://docs.oracle.com/javase/8/docs/technotes/guides/security/crypto/CryptoSpec.html#SigEx

Java Doc of Signature
https://docs.oracle.com/javase/7/docs/api/java/security/Signature.html

JAVA RSA Generate KeyPair, Encryption, Decryption and Performance Testing

Motivation

To try RSA Enc/Dec and check its performance.

Simple Note

Test file TestRSAUtils.java

Implementation RSAUtils.java

Performance testing TestRSAPerformance.java

Refactoring

Move AESUtils.encrypt and AESUtils.decrypt to CryptoUtils since they are no different from RSA encrypt/decrypt.

Still wrap them in AESUtils/RSAUtils for convenience.
(Can import less class) (Just one class less)

Hmm...


Performance Testing

Test over
PublicKey/PrivateKey, Encryption/Decryption,
Create Cipher each time/Reuse Cipher,
Key Size 1024/2048/4096/8192 bits
total 32 cases, 100 runs each

Results listed in the google sheet
JAVA RSA Performance Testing

Not very accurate but...

Summary

1. PublicKey Enc/Dec is faster than PrivateKey Enc/Dec

2. Key Size affects, the longer the slower.

3. Reuse Cipher similar to create Cipher each time, sometimes even slower.

Conclusions

1. Use proper Key Size, 2048 probably a good choice.

2. Can create Cipher each time, no need to cache it.

3. Can also cache and reuse Cipher, will slightly increase performance in most cases.

References

Java: How can I generate PrivateKey from a string?

Load RSA public key from file

JAVA AES Performance Testing

Motivation

Want to know the better choice of coding.

Simple Note

Test functions in
TestAESPerformance.java

Also use a simple helper class for performance testing
PerformanceTestHelper.java


Test Cases

100 runs each

1. testCreateInstanceEachTimeCommon: Simple AES with pre-generated key, no salt, create Cipher each time

2. testReuseInstanceCommon: Simple AES with pre-generated key, no salt, reuse Cipher

3. testCreateInstanceEachTimePBK: Use SecretKeyFactory and PBEKeySpec to generate SecretKey with PBKDF2WithHmacSHA256 algorithm, generate key and create Cipher each time

4. testReuseInstancePBK: Use SecretKeyFactory and PBEKeySpec to generate SecretKey with PBKDF2WithHmacSHA256 algorithm, reuse Cipher

Results

Summary

1. Use pre-generated key is much faster
(testCreateInstanceEachTimeCommon is much faster than testCreateInstanceEachTimePBK).

2. Reuse Cipher is much faster.

3. Once the Cipher is created, the speed of Encryption are similar between Common and PBK
(the 90% time of testReuseInstancePBK even faster than testReuseInstanceCommon)

Conclusion

Do not create Cipher each time, try to reuse it.

References

Round double
Round a double to 2 decimal places

List Initial Size (need to get the fastest 90% so...)
Initial size for the ArrayList

JAVA AES Encrypt Decrypt with Any Key, IV and Salt

Motivation

Convenient, that's all

Simple Note

Test File:
  TestAESImpl.java


Implementation:
  AESImpl.java


References

Java MessageDigest supported algorithms? MD5, SHA-1, SHA-256
Class MessageDigest

How to Dupe a char array
Does calling clone() on an array also clone its contents?

JAVA AES Encryption Decryption with IV, Salt and any length of Key

Simple Note

See the method TestEncryptDecryptWithKeyIvSalt in TestAESUtils.java

and getCipher(int mode, char[] key, byte[] ivBytes, byte[] saltBytes) in AESUtils.java

References

Java 256-bit AES Password-Based Encryption


Java AES and using my own Key

Java AES Key Generation, Encryption and Decryption

Simple Note

Test File:
  TestAESUtils.java

Utils Implementation
  AESUtils.java
  CryptoUtils.java

If you get Illegal key size error, probably need to download JCE Jars, please refer to

InvalidKeyException Illegal key size

and

Java Security: Illegal key size or default parameters?


Other References

The article teach me use char[] over String
Why is char[] preferred over String for passwords?

This one teach me basic Encryption and Decryption
Initial bytes incorrect after Java AES/CBC decryption

How to convert between char[] and byte[]
Converting char[] to byte[]

Use ExecutorService to Run Multiple Callable and Get Results In Order

Motivation

ExecutorService can run multiple Runnable/Callable, but there is no method for get their Results / Exception in order
(Similar to Promise.all in JavaScript)

So try to build a simple solution.

Simple Note

Test / Sample
  ConcurrentParallel.java
Implementation to Run Multiple Callable
  CallableParallelExecutor.java
Wrapper Class to wrap Callable
  CallableWrapper.java

JAVA Thread Debounce and Throttle

Simple Note

Most basic practice of Thread at MultiThread.java

Implement, test Debounce and Throttle Thread at MultiThreadDebounceThrottle.java, ThreadUtils.java