hubertf's NetBSD blog

hubertf's NetBSD Blog
Send interesting links to hubert at feyrer dot de!

[20130321]	Ansible, EC2 and NetBSD milestone 3 reached: Web and database in the cloud With the previous work on setting up a local VM as database and web server and setting up a Xen VM in Amazon's EC2 cloud combined, it is pretty straight forward to setup a EC2 instance that has all the software to serve a simple web application from the cloud. The single steps are: Prepare the environment with proper time, SSH agent and EC2 firewall groups Setup EC2 instance with pkgin and ansible Do basic preparations to meet our standards for logins, shells and general usability and security Setup database server with DB software, user and import of data Setup web server with all the software and some demo application The following details show all the commands can and their output in more detail: Make sure time is set properly - needed when talking to Amazon EC2: % sudo sh /etc/rc.d/ntpd stop ntpd not running? (check /var/run/ntpd.pid). % sudo sh /etc/rc.d/ntpdate restart Setting date via ntp. % sudo sh /etc/rc.d/ntpd start Starting ntpd. % date Sat Mar 16 16:46:19 CET 2013 Teach our EC2 SSH key to SSH agent, so we don't have to type a password (which we don't know anyways - EC2 only works with SSH keys): % ssh-add -l Could not open a connection to your authentication agent. % % eval `ssh-agent` Agent pid 10467 % ssh-add -l The agent has no identities. % ssh-add ../../euca2ools/key-eucaHF.pem Identity added: ../../euca2ools/key-eucaHF.pem (../../euca2ools/key-eucaHF.pem) % ssh-add -l 2048 d5:25:19:3d:59:40:35:32:03:f7:c5:83:de:19:b6:d0 ../../euca2ools/key-eucaHF.pem (RSA) Check security (firewall) groups - those are stored in EC2, and we have previously set them up: % euca-describe-groups ... GROUP sg-a854b3c3 749335780469 ec2-webservers Web servers PERMISSION 749335780469 ec2-webservers ALLOWS tcp 22 22 FROM CIDR 0.0.0.0/0 PERMISSION 749335780469 ec2-webservers ALLOWS tcp 80 80 FROM CIDR 0.0.0.0/0 PERMISSION 749335780469 ec2-webservers ALLOWS icmp -1 -1 FROM CIDR 0.0.0.0/0 See if there are any EC2 instances running: % euca-describe-instances % No - that's fine, we are about to change that! Run first playbook to launch EC2 instance and prepare it for using with ansible: % ansible-playbook -i hosts-HF config-ec2-prepare1vm.yml PLAY [localhost] ******************* TASK: [Launch new EC2 instance] ***************** changed: [127.0.0.1] TASK: [Give the system 30 seconds to boot up] ***************** changed: [127.0.0.1] TASK: [Get rid of SSH "Are you sure you want to continue connecting (yes/no)?" query] ***************** changed: [127.0.0.1] TASK: [Fix /usr/bootstrap.sh to run pkgin with -y] ***************** changed: [127.0.0.1] => (item={'cmd': 'install /usr/bootstrap.sh /usr/bootstrap.sh.orig'}) changed: [127.0.0.1] => (item={'cmd': 'chmod +w /usr/bootstrap.sh'}) changed: [127.0.0.1] => (item={'cmd': 'sed "s,bin/pkgin update,bin/pkgin -y update," /usr/bootstrap.sh'}) changed: [127.0.0.1] => (item={'cmd': 'chmod -w /usr/bootstrap.sh'}) TASK: [Install pkgin via /usr/bootstrap.sh] ***************** changed: [127.0.0.1] => (item={'cmd': u'env PATH=/usr/sbin:${PATH} /usr/bootstrap.sh binpkg'}) TASK: [Copy over Ansible binary package] ***************** changed: [127.0.0.1] TASK: [Install Ansible dependencies] ***************** changed: [127.0.0.1] TASK: [Install Ansible package (manually)] ***************** changed: [127.0.0.1] TASK: [Setup lame /usr/bin/python symlink] ***************** changed: [127.0.0.1] PLAY RECAP ***************** 127.0.0.1 : ok=9 changed=9 unreachable=0 failed=0 We now have a EC2 instance running that has Ansible installed: % euca-describe-instances RESERVATION r-d77272ad 749335780469 ec2-webservers INSTANCE i-9fafc2f2 ami-5d0f8034 ec2-107-22-69-112.compute-1.amazonaws.com ... With this EC2 instance, we can do some basic preparations for our standards, e.g. a login without requiring root (and while there, actually disable allowing as root), setup sudo and a proper shell: % env ANSIBLE_HOSTS=./ec2.py ansible-playbook config-ec2-basic.yml PLAY [security_group_ec2-webservers] ***************** TASK: [ping] ***************** ok: [ec2-107-22-69-112.compute-1.amazonaws.com] TASK: [Install tcsh] ***************** changed: [ec2-107-22-69-112.compute-1.amazonaws.com] TASK: [Add user feyrer] ***************** changed: [ec2-107-22-69-112.compute-1.amazonaws.com] TASK: [Create ~feyrer/.ssh directory] ***************** changed: [ec2-107-22-69-112.compute-1.amazonaws.com] TASK: [Enable ssh login with ssh-key] ***************** changed: [ec2-107-22-69-112.compute-1.amazonaws.com] TASK: [Install sudo] ***************** changed: [ec2-107-22-69-112.compute-1.amazonaws.com] TASK: [Enable PW-less sudo-access for everyone in group 'wheel'] ***************** changed: [ec2-107-22-69-112.compute-1.amazonaws.com] TASK: [Disable ssh logins as root] ***************** ok: [ec2-107-22-69-112.compute-1.amazonaws.com] PLAY RECAP ***************** ec2-107-22-69-112.compute-1.amazonaws.com : ok=8 changed=6 unreachable=0 failed=0 Let's have a look if things actually work: % ssh 107.22.69.112 id uid=1000(feyrer) gid=100(users) groups=100(users),0(wheel) % ssh ec2-107-22-69-112.compute-1.amazonaws.com id uid=1000(feyrer) gid=100(users) groups=100(users),0(wheel) % ssh ec2-107-22-69-112.compute-1.amazonaws.com sudo id uid=0(root) gid=0(wheel) groups=0(wheel),2(kmem),3(sys),4(tty),5(operator),20(staff),31(guest) Next, install database software and import our demo database, just as we did in out local VM: % env ANSIBLE_HOSTS=./ec2.py ansible-playbook config-ec2-dbserver.yml PLAY [security_group_ec2-webservers] ***************** TASK: [Install mysql] ***************** changed: [ec2-107-22-69-112.compute-1.amazonaws.com] TASK: [Install MySQL rc.d script] ***************** changed: [ec2-107-22-69-112.compute-1.amazonaws.com] TASK: [Start MySQL service] ***************** changed: [ec2-107-22-69-112.compute-1.amazonaws.com] TASK: [Install python-mysqldb (for mysql_user module)] ***************** changed: [ec2-107-22-69-112.compute-1.amazonaws.com] TASK: [Setup DB] ***************** changed: [ec2-107-22-69-112.compute-1.amazonaws.com] TASK: [Add db-user] ***************** changed: [ec2-107-22-69-112.compute-1.amazonaws.com] TASK: [Copy over DB template] ***************** changed: [ec2-107-22-69-112.compute-1.amazonaws.com] TASK: [Import DB data] ***************** changed: [ec2-107-22-69-112.compute-1.amazonaws.com] PLAY RECAP ***************** ec2-107-22-69-112.compute-1.amazonaws.com : ok=8 changed=8 unreachable=0 failed=0 Again, let's see if everything works as expected: % ssh ec2-107-22-69-112.compute-1.amazonaws.com ... ip-10-202-65-196: {1} mysql -u webapp -p webapp Enter password: ** ... mysql> show tables; +------------------+ \| Tables_in_webapp \| +------------------+ \| names \| +------------------+ 1 row in set (0.00 sec) mysql> select * from names; +----+--------+------+ \| id \| first \| last \| +----+--------+------+ \| 1 \| Donald \| Duck \| \| 2 \| Daisy \| Duck \| +----+--------+------+ 2 rows in set (0.00 sec) mysql> exit Bye ip-10-202-65-196: {2} exit logout Connection to ec2-107-22-69-112.compute-1.amazonaws.com closed. Last, add Apache+PHP and our small demo web-application: % env ANSIBLE_HOSTS=./ec2.py ansible-playbook config-ec2-webserver.yml PLAY [security_group_ec2-webservers] ***************** TASK: [Installing ap24-php53 package and dependencies] ***************** changed: [ec2-107-22-69-112.compute-1.amazonaws.com] TASK: [Install Apache rc.d script] ***************** changed: [ec2-107-22-69-112.compute-1.amazonaws.com] TASK: [Enable and start Apache service] ***************** changed: [ec2-107-22-69-112.compute-1.amazonaws.com] TASK: [Enable PHP in Apache config file] ******************* changed: [ec2-107-22-69-112.compute-1.amazonaws.com] => (item={'re': 'LoadModule.mod_php5.so', 'l': 'LoadModule php5_module lib/httpd/mod_php5.so'}) changed: [ec2-107-22-69-112.compute-1.amazonaws.com] => (item={'re': 'AddHandler.x-httpd-php', 'l': 'AddHandler application/x-httpd-php .php'}) TASK: [Make Apache read index.php] ******************* changed: [ec2-107-22-69-112.compute-1.amazonaws.com] TASK: [Add simple PHP test - see http://10.0.0.181/phptest.php] ***************** changed: [ec2-107-22-69-112.compute-1.amazonaws.com] TASK: [Install phpmyadmin] ***************** changed: [ec2-107-22-69-112.compute-1.amazonaws.com] TASK: [Enable phpmyadmin in Apache config] ***************** changed: [ec2-107-22-69-112.compute-1.amazonaws.com] TASK: [Fix Apache access control for phpmyadmin] ***************** changed: [ec2-107-22-69-112.compute-1.amazonaws.com] TASK: [Enable PHP modules in PHP config file] ******************* changed: [ec2-107-22-69-112.compute-1.amazonaws.com] => (item={'re': '^extension.zlib.so', 'l': 'extension=zlib.so'}) changed: [ec2-107-22-69-112.compute-1.amazonaws.com] => (item={'re': '^extension.zip.so', 'l': 'extension=zip.so'}) changed: [ec2-107-22-69-112.compute-1.amazonaws.com] => (item={'re': '^extension.mysqli.so', 'l': 'extension=mysqli.so'}) changed: [ec2-107-22-69-112.compute-1.amazonaws.com] => (item={'re': '^extension.mysql.so', 'l': 'extension=mysql.so'}) changed: [ec2-107-22-69-112.compute-1.amazonaws.com] => (item={'re': '^extension.mcrypt.so', 'l': 'extension=mcrypt.so'}) changed: [ec2-107-22-69-112.compute-1.amazonaws.com] => (item={'re': '^extension.mbstring.so', 'l': 'extension=mbstring.so'}) changed: [ec2-107-22-69-112.compute-1.amazonaws.com] => (item={'re': '^extension.json.so', 'l': 'extension=json.so'}) changed: [ec2-107-22-69-112.compute-1.amazonaws.com] => (item={'re': '^extension.gd.so', 'l': 'extension=gd.so'}) changed: [ec2-107-22-69-112.compute-1.amazonaws.com] => (item={'re': '^extension.gettext.so', 'l': 'extension=gettext.so'}) changed: [ec2-107-22-69-112.compute-1.amazonaws.com] => (item={'re': '^extension.bz2.so', 'l': 'extension=bz2.so'}) TASK: [Create directory for webapp] ******************* changed: [ec2-107-22-69-112.compute-1.amazonaws.com] TASK: [Deploy example webapp] ***************** changed: [ec2-107-22-69-112.compute-1.amazonaws.com] TASK: [Create webapp symlink for easy access] ***************** changed: [ec2-107-22-69-112.compute-1.amazonaws.com] NOTIFIED: [restart apache] ***************** changed: [ec2-107-22-69-112.compute-1.amazonaws.com] PLAY RECAP ***************** ec2-107-22-69-112.compute-1.amazonaws.com : ok=14 changed=14 unreachable=0 failed=0 Test! % links -dump http://ec2-107-22-69-112.compute-1.amazonaws.com/ It works! % links -dump http://ec2-107-22-69-112.compute-1.amazonaws.com/phptest.php PHP Logo PHP Version 5.3.17 System NetBSD ip-10-202-65-196.ec2.internal 6.0.1 NetBSD 6.0.1 (XEN3PAE_DOMU) i386 Build Date Dec 14 2012 10:31:13 ... % links -dump http://ec2-107-22-69-112.compute-1.amazonaws.com/webapp/ Showing table hf.names: +--------------------+ \| id \| first \| last \| \|----+--------+------\| \| 1 \| Donald \| Duck \| \|----+--------+------\| \| 2 \| Daisy \| Duck \| +--------------------+ ---------------------------------------------------------------------- Enter new values: first: _____________________ last: _____________________ [ Submit ] At this point, everything is setup and can be enjoyed. If the instance is needed no longer, it can be terminated: % euca-describe-instances RESERVATION r-d77272ad 749335780469 ec2-webservers INSTANCE i-9fafc2f2 ami-5d0f8034 ec2-107-22-69-112.compute-1.amazonaws.com ... % euca-terminate-instances i-9fafc2f2 INSTANCE i-9fafc2f2 % euca-describe-instances RESERVATION r-d77272ad 749335780469 ec2-webservers INSTANCE i-9fafc2f2 ami-5d0f8034 terminated eucaHF ... What's next on my journey is to put database and webserver into separate VMs. First one of each, and then see if I find the nerve to look into a setup with more redundancy. Shameless plug:** I'll talk about the ansible and euca2ools packages at pkgsrcCon 2013 in Berlin. Join in if you're curious about what the actual playbooks used in the above examples look like! [Tags: amazon, ansible, apache, ec2, mysql, php, xen]
[20130228]	Ansible and NetBSD milestone 1 reached: playbooks for system config, web+db servers In my quest to play with Ansible, I've reached my first milestone: I now have playbooks that take a basic NetBSD installation, configure it into a usable base installation, and then add a MySQL database, Apache and PHP to use it as webserver, and then deploy a simple web application. The playbooks are too emberassing to publish, but here are the steps to get things going: Setup NetBSD 6.0 with "base" and "etc" set, also add "pkgin" from menu Allow root logins via ssh (for a start, will be changed later) Install ansible-1.0nb1 binary package with all its depends From a management station, run: ansible-playbook -k -i hosts-HF config-netbsd-basic.yml Then, run: ansible-playbook -i hosts-HF config-netbsd-dbserver.yml Last, run: ansible-playbook -i hosts-HF config-netbsd-webserver.yml After that, a simple "phptest()" page, phpmyadmin and my simple PHP-based web application can be run. Administration of the system is via SSH and sudo, root logins were disabled in the first ansible playbook. Now to tweak the ansible playbooks to look less ugly, use variables, and then separate database and webserver into two separate machines - all in preparation to move them into the Amazon EC2 cloud. Stay tuned! For the record, here's a log of the three ansible playbooks above, starting from my basic NetBSD installation that already has pkgin and ansible: % ansible-playbook -k -i hosts-HF config-netbsd-basic.yml SSH password: PLAY [netbsd] ******************* GATHERING FACTS ***************** ok: [10.0.0.181] TASK: [Install tcsh] ***************** changed: [10.0.0.181] TASK: [Add user feyrer] ***************** changed: [10.0.0.181] TASK: [Create ~feyrer/.ssh directory] ***************** changed: [10.0.0.181] TASK: [Enable ssh login with ssh-key] ***************** changed: [10.0.0.181] TASK: [Install sudo] ***************** changed: [10.0.0.181] TASK: [Enable PW-less sudo-access for everyone in group 'wheel'] ***************** changed: [10.0.0.181] TASK: [Disable ssh logins as root] ***************** changed: [10.0.0.181] NOTIFIED: [restart sshd] ***************** changed: [10.0.0.181] PLAY RECAP ***************** 10.0.0.181 : ok=9 changed=8 unreachable=0 failed=0 % ansible-playbook -i hosts-HF config-netbsd-dbserver.yml PLAY [dbservers] ***************** GATHERING FACTS ***************** ok: [10.0.0.181] TASK: [Install mysql] ***************** changed: [10.0.0.181] TASK: [Install MySQL rc.d script] ***************** changed: [10.0.0.181] TASK: [Start MySQL service] ***************** changed: [10.0.0.181] TASK: [Install python-mysqldb (for mysql_user module)] ***************** changed: [10.0.0.181] TASK: [Setup DB] ***************** changed: [10.0.0.181] TASK: [Add db-user] ***************** changed: [10.0.0.181] TASK: [Copy over DB template] ***************** changed: [10.0.0.181] TASK: [Import DB data] ***************** changed: [10.0.0.181] PLAY RECAP ***************** 10.0.0.181 : ok=9 changed=8 unreachable=0 failed=0 %% ansible-playbook -i hosts-HF config-netbsd-webserver.yml PLAY [webservers] ***************** GATHERING FACTS ***************** ok: [10.0.0.181] TASK: [Installing ap24-php53 package and dependencies] ***************** changed: [10.0.0.181] TASK: [Install Apache rc.d script] ***************** changed: [10.0.0.181] TASK: [Enable and start Apache service] ***************** changed: [10.0.0.181] TASK: [Enable PHP in Apache config file] ******************* changed: [10.0.0.181] => (item={'re': 'LoadModule.mod_php5.so', 'l': 'LoadModule php5_module lib/httpd/mod_php5.so'}) changed: [10.0.0.181] => (item={'re': 'AddHandler.x-httpd-php', 'l': 'AddHandler application/x-httpd-php .php'}) TASK: [Make Apache read index.php] ******************* changed: [10.0.0.181] TASK: [Add simple PHP test - see http://10.0.0.181/phptest.php] ***************** changed: [10.0.0.181] TASK: [Install phpmyadmin] ***************** changed: [10.0.0.181] TASK: [Enable phpmyadmin in Apache config] ***************** changed: [10.0.0.181] TASK: [Enable PHP modules in PHP config file] ******************* changed: [10.0.0.181] => (item={'re': '^extension.zlib.so', 'l': 'extension=zlib.so'}) changed: [10.0.0.181] => (item={'re': '^extension.zip.so', 'l': 'extension=zip.so'}) changed: [10.0.0.181] => (item={'re': '^extension.mysqli.so', 'l': 'extension=mysqli.so'}) changed: [10.0.0.181] => (item={'re': '^extension.mysql.so', 'l': 'extension=mysql.so'}) changed: [10.0.0.181] => (item={'re': '^extension.mcrypt.so', 'l': 'extension=mcrypt.so'}) changed: [10.0.0.181] => (item={'re': '^extension.mbstring.so', 'l': 'extension=mbstring.so'}) changed: [10.0.0.181] => (item={'re': '^extension.json.so', 'l': 'extension=json.so'}) changed: [10.0.0.181] => (item={'re': '^extension.gd.so', 'l': 'extension=gd.so'}) changed: [10.0.0.181] => (item={'re': '^extension.gettext.so', 'l': 'extension=gettext.so'}) changed: [10.0.0.181] => (item={'re': '^extension.bz2.so', 'l': 'extension=bz2.so'}) TASK: [Fix Apache access control] ******************* changed: [10.0.0.181] TASK: [Create directory for webapp] ***************** changed: [10.0.0.181] TASK: [Deploy example webapp] ***************** changed: [10.0.0.181] TASK: [Create webapp symlink for easy access] ***************** changed: [10.0.0.181] NOTIFIED: [restart apache] ***************** changed: [10.0.0.181] PLAY RECAP ***************** 10.0.0.181 : ok=15 changed=14 unreachable=0 failed=0 % links -dump http://10.0.0.181/webapp/** Showing table hf.names: +--------------------+ \| id \| first \| last \| \|----+--------+------\| \| 1 \| Donald \| Duck \| \|----+--------+------\| \| 2 \| Daisy \| Duck \| +--------------------+ ---------------------------------------------------------------------- Enter new values: first: _____________________ last: _____________________ [ Submit ] % [Tags: amazon, ansible, apache, ec2, mysql, php]
[20071001]	Performance improvements: numbers and pictures Andrew Doran, who was recently hired by the NetBSD project to work on NetBSD's SMP implementation, has done a lot of good work, and he has merged some of his work from the vmlocking-branch into NetBSD-current. Effects of this are that time for build.sh on a quad-Opteron went down by ~10%. Andrew also updated his previous benchmarks, and posted about his recent results: ``Most of the sysbench runs that I've seen to date have sysbench running on the same machine as the database. That's a good test but with the exception of small installations and out-of-band activity, production setups rarely look like that. So I ran sysbench itself on a seperate dual core system.'' There are images that compare NetBSD 3 with NetBSD-current (where most of Andrew's changes are now), and NetBSD-current compared to Linux, FreeBSD, and OpenBSD: The original benchmarks didn't include Solaris/x86, so Jaime Fournier sat down and repeated the test (on a single system). The results show that NetBSD beats Solaris by ~25% in the ReadOnly test, and that they're about on par in the ReadWrite test, with NetBSD kicking in earlier WRT the number of client threads, but Solaris keeping up longer before they both degrade. The courves are quite similar, and my guess is that there is some room for finetuning there: [Tags: benchmark, mysql, performance]
[20050307]	Quadrupling MySQL performance In light of the recent MySQL benchmark, Chuck Silvers has posted his analysis of the situation as well as ways to increase performance from about 3 transactions per second to about 12 TPS. Some of the tuning is available today in NetBSD-current (and will be in NetBSD 3.0), some other changes need more thought. Of course tuning a complex application like a database engine will always require some manual work combined with understanding of the application, but NetBSD's there to help do the job. [Tags: benchmark, mysql]
[20050209]	Article: Comparing MySQL performance (Part II, the results, Update 1) In the second part of the comparison of MySQL performance on various operating systems, NetBSD performs quite well on a number of benchmarks with 1 CPU (even beating Linux in one case), but loses badly in SMP environment. I wonder if this is impacted by the fact that threaded applications use only 1 CPU on NetBSD, even in SMP environments - one has to set the "PTHREAD_CONCURRENCY" environment variable to something between 1 and the number of CPUs to improve concurrency. Update: Check this out for a comparison of different DB engines (on Windows, but still; too bad it doesn't include Postgres!). [Tags: Articles, benchmark, mysql]
[20050208]	Teaser: Using MySQL to benchmark OS performance (Part I) After the recent fuzz about "real" vs. microbenchmarks, someone apparently sat down to use MySQL to benchmark several operating systems, including NetBSD, FreeBSD, OpenBSD, Solaris and Linux. This first article describes the setup of the systems and the benchmarks used to measure performance. The results will be in the next article, so we can stay tuned! [Tags: benchmark, mysql]