Saturday, October 19, 2013

Grails database-migration-plugin: DB independent diff files

If you are using Grails database-migration-plugin and your application has to support MySQL as well as Oracle, you have 2 choices currently. As the underlying Liquibase library is currently unable to create real database-agnostic migration files when performing a diff, you can:

  • create 2 different sets of migration files, one for MySQL one for Oracle. Drawback of this is, that this is error prone and anything else than DRY.
  • Convert the created migration files automatically so they are real database agnostic.
Thanks to the Grails database-migration-plugin hooks (when using database-migration plugin version >= 1.3), we can do that automatically on initial start after creating a new migration file. Migration files are only migrated once, and migrated files will be marked with a special comment to indicate conversion.

In changelog.groovy, define all types you want to use for Oracle and MySQL (you can extend that to support other db types, easily):

databaseChangeLog = {
  
  /*
    DATABASE SPECIFIC TYPE PROPERTIES
   */
  property name: "text.type", value: "varchar(50)", dbms: "mysql"
  property name: "text.type", value: "varchar2(500)", dbms: "oracle"

  property name: "string.type", value: "varchar", dbms: "mysql"
  property name: "string.type", value: "varchar2", dbms: "oracle"

  property name: "boolean.type", value: "bit", dbms: "mysql"
  property name: "boolean.type", value: "number(1,0)", dbms: "oracle"

  property name: "int.type", value: "bigint", dbms: "mysql"
  property name: "int.type", value: "number(19,0)", dbms: "oracle"

  property name: "clob.type", value: "longtext", dbms: "mysql"
  property name: "clob.type", value: "clob", dbms: "oracle"

  property name: "blob.type", value: "longblob", dbms: "mysql"
  property name: "blob.type", value: "blob", dbms: "oracle"

  /* DATABASE SPECIFIC FEATURES */
  property name: "autoIncrement", value: "true", dbms: "mysql"
  property name: "autoIncrement", value: "false", dbms: "oracle"


  /* Database specific prerequisite patches */
  changeSet(author: "changelog", id: "ORACLE-PRE-1", dbms: "oracle") {
    createSequence(sequenceName: "hibernate_sequence")
  }

  /* Patch files */  
  include file: 'initial.groovy'

}

Then create a Callback Bean class for database-migration-plugin and register it in resources.groovy:

migrationCallbacks(DbmCallbacks)

Bean:

import liquibase.Liquibase
import liquibase.database.Database
import org.codehaus.groovy.grails.plugins.support.aware.GrailsApplicationAware;
import org.codehaus.groovy.grails.commons.GrailsApplication

class DbmCallbacks implements GrailsApplicationAware {
  private static final String MIGRATION_KEY = "AUTO_REWORKED_MIGRATION_KEY"
  private static final String MIGRATION_HEADER = "*/ ${MIGRATION_KEY} */"
  // DB-Specific types to liquibase properties mapping
  // see changelog.groovy for defined liquibase properties
  Map<String,String> liquibaseTypesMapping = [
          // start with specific ones, then unspecific ones.
          'type: "varchar(50)"': "type: '\\\${text.type}'",
          'type: "varchar2(500)"': "type: '\\\${text.type}'",
          'type: "varchar"': "type: '\\\${string.type}'",
          'type: "varchar2"': "type: '\\\${string.type}'",
          'type: "bit"': "type: '\\\${boolean.type}\'",
          'type: "number(1,0)"': "type: '\\\${boolean.type}'",
          'type: "bigint"': "type: '\\\${int.type}'",
          'type: "number(19,0)"': "type: '\\\${int.type}'",
          'type: "longtext"': "type: '\\\${clob.type}\'",
          'type: "clob"': "type: '\\\${clob.type}\'",
          'type: "longblob"': "type: '\\\${blob.type}\'",
          'type: "blob"': "type: '\\\${blob.type}\'",
          // regEx (e.g. "varchar(2)" to ${string.type}(2)'. Do not add trailing "'", here!
          '/.*(type: "varchar\\((.*)\\)").*/': "type: '\\\${string.type}",
          '/.*(type: "varchar2\\((.*)\\)").*/': "type: '\\\${string.type}",
          // db features
          'autoIncrement: "true"': "autoIncrement: '\\\${autoIncrement}'"
  ]

 void beforeStartMigration(Database database) {
   reworkMigrationFiles()
 }
 private void reworkMigrationFiles() {
    def config = grailsApplication.config.grails.plugin.databasemigration
    def changelogLocation = config.changelogLocation ?: 'grails-app/migrations'
    new File(changelogLocation)?.listFiles().each { File it ->
      List updateOnStartFileNames = config.updateOnStartFileNames
      if (updateOnStartFileNames?.contains(it.name)) {
        // do not convert updateOnStart files.
        return
      }
      convertMigrationFile(it)
    }
  }
 private void convertMigrationFile(File migrationFile) {
    def content = migrationFile.text
    if (content.contains(MIGRATION_KEY)) return
    liquibaseTypesMapping.each {
      String pattern = it.key
      String replace = it.value
      if (pattern.startsWith('/')) {
        // Handle regex pattern.
        def newContent = new StringBuffer()
        content.eachLine { String line ->
          def regEx = pattern[1..-2] // remove leading and trailing "/"
          def matcher = (line =~ regEx)
          if (matcher.matches() && matcher.groupCount() == 2) {
              String replaceFind = matcher[0][1] // this is the found string, e.g. 'type: "varchar(22)"'
              String replacement = "${replace}(${matcher[0][2]})\'"  // new string, e.g. "type: '${string.type}(22)' "
              line = line.replace(replaceFind, replacement)
          }
          newContent += "${line}\n"
        }
        content = newContent
      } else {
        // non-regEx, so replace all in one go.
        content = content.replaceAll(pattern, replace)
      }
    }
    // mark file as already migrated
    content = "${MIGRATION_HEADER} +"\n"+ content
    migrationFile.write(content, 'UTF-8')
    log.warn "*** Converted database migration file ${migrationFile.name} to be database independent"
  }


This for sure can be optimized (e.g. use only regEx definitions in the map and handle if no matcher groups are found, but it does it's job. 

Tested with MySQL and Oracle 11.0.2 XE.


Building 64bit TrueCrypt for OSX

Currently, TrueCrypt binaries are only available for PPC and i386 without any hardware accelleration.
Also, the available binaries are currently under suspect, as nobody knows if they were compiled from the official source code or if they were tampered by someone. (hick..).

A project tries to get funded to audit the TrueCrypt sources and binaries for any hidden backdoors: http://istruecryptauditedyet.com. The german C't magazine tried to rebuild the Windows binaries from the source code and found some suspect differences while comparing the binaries. See here [english translation] [original article in german].

To ensure at least you do not use tampered binaries, you can use this script to generate a 64bit OSX version from the TrueCrypt sources with hardware accellerated AES functions yourself. (Idea and patches see this Blog post).


#!/bin/sh
# Build TrueCrypt on OSX with 64bit and HW acc. AES
# 2013 http://roosbertl.blogspot.com
####
version=7.1a
md5="102d9652681db11c813610882332ae48"
sourcename="TrueCrypt ${version} Source.tar.gz"
####
download_filename="TrueCrypt%20${version}%20Source.tar.gz"
which /opt/local/bin/port &>/dev/null
if [ $? != 0 ]; then
echo "Port seems not to be installed."
echo "Please install www.macports.org, first" 
exit 1
fi
currDir=`pwd`
workDir="$0.$$"
echo "Creating TrueCrypt $version"
mkdir $workDir
trap "echo cleaning up; cd $currDir; rm -rf $workDir ; exit" SIGHUP SIGINT SIGTERM
echo "Getting required Ports.."
sudo port install wxWidgets-3.0 fuse4x nasm wget pkgconfig
sudo port select wxWidgets wxWidgets-3.0
echo " "
echo "Downloading $sourcename"
wget --quiet http://cyberside.planet.ee/truecrypt/$download_filename
echo "Checking md5.."
thisMd5=`openssl md5 < $sourcename | cut -d " " -f 2`
if [ ! "$md5" = "$thisMd5" ]; then
echo "MD5 checksum $thisMd5 does not match expected MD5 checksum $md5"
echo "Either the source file was modified or you tried to download a different version"
echo "FATAL ERROR. Aborting."
exit 1
else
echo "Checksum is ok."
fi
echo "Extracting '$sourcename'"
tar zxf "$sourcename"
cd truecrypt-${version}-source
echo "Getting Patch file.."
wget --quiet http://www.nerdenmeister.org/truecrypt-osx.patch
mkdir Pkcs11
cd Pkcs11
echo "Getting pkcs11 headers.."
wget --quiet ftp://ftp.rsasecurity.com/pub/pkcs/pkcs-11/v2-20/pkcs11.h
wget --quiet ftp://ftp.rsasecurity.com/pub/pkcs/pkcs-11/v2-20/pkcs11f.h
wget --quiet ftp://ftp.rsasecurity.com/pub/pkcs/pkcs-11/v2-20/pkcs11t.h
cd ..
echo "Patching TrueCrypt for 64bit and HW accellerated AES.."
patch -p0 < truecrypt-osx.patch
echo "Compiling..."
make -j4
echo "Compile done."
mv Main/TrueCrypt.app ..
echo "Cleanup.."
cd $currDir
rm -rf $0.$$
echo "Done creating TrueCrypt.app Version: $version"
# end





Wednesday, July 31, 2013

jMeter-Server on OSX

If you want to run a jmeter-server unattended on one or several OSX boxes, you can perform this:

1. create /Library/LaunchAgents/org.apache.jmeter.server.plist


#>sudo vi /Library/LaunchAgents/org.apache.jmeter.server.plist


<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>LimitLoadToSessionType</key>
<string>System</string>
<key>KeepAlive</key>
<true/>
<key>Label</key>
<string>org.apache.jmeter.server.plist</string>
<key>Program</key>
<string>/Applications/JMeter-2.9.app/Contents/Resources/bin/jmeter-server</string>
<key>WorkingDirectory</key>
<string>/var/log</string>
<key>RunAtLoad</key>
<true/>
</dict>
</plist>

Program path is  the path to the jmeter-server script. In the example above, I bundled jmeter 2.2 with Jar Bundler into an OSX app, added all jmeter folders to Contents/Resources (bin, lib folders) so I start the jmeter-server from the bundle app on several remote OSX boxes.

2. Load the plist file in launchctl:



# sudo launctl load /Library/LaunchAgents/org.apache.jmeter.server.plist

This should immediately start the jmeter-server with working directory set to /var/log (to get the jmeter-server.log logged in the system log dir)

3. Register remote jmeter-servers in jMeter

To register the jmeter-server instances in your local jMeter program, edit bin/jmeter.properties and edit the property "remote_hosts". Add your remote jmeter-servers by comma-separating the IP adresses. Example:

remote_hosts=127.0.0.1,192.168.17.12

Thursday, June 13, 2013

Creating OSX FusionDrive with Recovery Partition

Today I received several 2010 model iMacs which were upgraded with an additional 3rd party SSD by an Apple reseller. The reseller created a FusionDrive by using the HDD and the SSD.

After reception, I recognized that no Recovery Partition was available. The whole disks were occupied by the CoreStorage volume.

Almost every instruction I found on the web for creating a FusionDrive was without also preserving a Recovery Partition. So I rebuilt the FusionDrive with a working Recovery Partition on my own to be as most compliant to the default stock Apple Fusion Drive configuration you get on a Mac with a preconfigured Fusion Drive. (Original Apple partitioning of a 27" late 2012 iMac Fusion Drive see end of this article).

Warning: This procedure is deleting your data from the disks! Use on own risk.
Note: Take a backup of all your data before proceeding, as all the data will be wiped. I take backups using TimeMachine and by using CarbonCopyCloner.

Prerequistes:

  • Install CarbonCopyCloner
  •  CarbonCopyCloner clone of your internal HDD to an external USB HDD (we will boot that later). If you receive a warning that no Recovery HD exists on the target USB drive, open the CarbonCopyCloner Windows > Disk Utility > Recovery HD > Select your USB drive and clone the Recovery Partition onto the USB drive.

Did I mention to take a TimeMachine Backup as well? Do that to be on the safe side.

After you made your backups, proceed with this steps:

1. Boot from your CarbonCopy USB clone by pressing the option (ALT) key during power up (or boot the recovery partition using CMD-R key after power on).
2. Start a Terminal
3. Check your current disk partitions and CoreStorage setup:

# sudo diskutil cs list 
CoreStorage logical volume groups (1 found)

|
+-- Logical Volume Group 78E316BB-911C-4456-9128-6925CDC3AE5F
    =========================================================
    Name:         FusionDrive
    Status:       Online
    Size:         1127552614400 B (1.1 TB)
    Free Space:   19023224832 B (19.0 GB)
    |
    +-< Physical Volume A3B20C13-4576-4FDF-A40D-F23BAA493C4C
    |   ----------------------------------------------------
    |   Index:    0
    |   Disk:     disk0s2
    |   Status:   Online
    |   Size:     127691702272 B (127.7 GB)
    |
    +-< Physical Volume 2E75ED2E-909F-44AF-A58E-57F94ABAD85C
    |   ----------------------------------------------------
    |   Index:    1
    |   Disk:     disk1s2
    |   Status:   Online
    |   Size:     999860912128 B (999.9 GB)
    |
    +-> Logical Volume Family FE296C1B-9152-42FE-8C6A-40DE18D747FA
        ----------------------------------------------------------
        Encryption Status:       Unlocked
        Encryption Type:         None
        Conversion Status:       NoConversion
        Conversion Direction:    -none-
        Has Encrypted Extents:   No
        Fully Secure:            No
        Passphrase Required:     No
        |
        +-> Logical Volume 2B4753CB-7D8C-4E57-BA81-C643AE84BF4F
            ---------------------------------------------------
            Disk:               disk2
            Status:             Online
            Size (Total):       1100000002048 B (1.1 TB)
            Size (Converted):   -none-
            Revertible:         No
            LV Name:            Macintosh HD
            Volume Name:        Macintosh HD
            Content Hint:       Apple_HFS


4. Note the UUID identifier of the Logical Volume group (marked in red)
5. Split up the existing FusionDrive CoreStorageVolume. If you do not have a CoreStorage volume set up, you can skip this step

# sudo diskutil cs delete <YOUR_UUID>, example:
# sudo diskutil cs delete 78E316BB-911C-4456-9128-6925CDC3AE5F

6. Format the internal HDD using Disk Utility
7. Start CarbonCopyCloner, then open Window > Hard Disk Management. Tab on "Recovery HD", select your internal HDD volume and click on the Create Recovery-HD partition button.
8. Now it is time to create your CoreStorage Logical Volume Group. But in contrast to many instructions on the net, we will not use the whole internal HDD, but only the free partition on the HDD! It is also important to state the SSD drive as the first disk to have optimum speed.
9. Check your current paritioning:

# diskutil list
/dev/disk0
   #:                       TYPE NAME        SIZE     IDENTIFIER
   0:      GUID_partition_scheme             *128.0GB disk0
   1:                        EFI             209.7 MB disk0s1
   2:                  Apple_HFS Untitled    127.7 GB disk0s2
/dev/disk1
   #:                       TYPE NAME        SIZE     IDENTIFIER
   0:      GUID_partition_scheme             *1.0 TB  disk1
   1:                        EFI             209.7 MB disk1s1
   2:                  Apple_HFS hdd         999.2 GB disk1s2
   3:                 Apple_Boot Recovery HD 784.2 MB disk1s3

/dev/disk3
   #:                       TYPE NAME        SIZE     IDENTIFIER
   0:                  Apple_HFS CarbonCopy  *998.7GB disk4

The partition disk1s2 is the free partition on the internal disk we will use for the FusionDrive.
The partition disk1s3 is the newly created Recovery Partition.

10. Create a new CoreStorage Volume. Disk0 in this example is the SSD drive.

# sudo diskutil cs create FusionDrive disk0 disk1s2
Password:
Started CoreStorage operation
Unmounting disk0
Repartitioning disk0
Unmounting disk
Creating the partition map
Rediscovering disk0
Adding disk0s2 to Logical Volume Group
Unmounting disk1s2
Touching partition type on disk1s2
Adding disk1s2 to Logical Volume Group
Creating Core Storage Logical Volume Group
Switching disk0s2 to Core Storage
Switching disk1s2 to Core Storage
Waiting for Logical Volume Group to appear
Discovered new Logical Volume Group "71377E10-7126-4E7B-A52D-F96F383F56BA"
Core Storage LVG UUID: 71377E10-7126-4E7B-A52D-F96F383F56BA
Finished CoreStorage operation

11. Now create the CoreStorage Logical Volume:
Note the LVG UUID printed by the command in step 10 (marked in red) and use that id:

diskutil cs createVolume <YOUR_LVG_UUID> jhfs+ "Macintosh HD" 100%

Example:
diskutil cs createVolume 71377E10-7126-4E7B-A52D-F96F383F56BA jhfs+ "Macintosh HD" 100%

Started CoreStorage operation
Waiting for Logical Volume to appear
Formatting file system for Logical Volume
Initialized /dev/rdisk6 as a 1 TB HFS Plus volume with a 90112k journal
Mounting disk
Core Storage LV UUID: 04085D2E-D630-4EED-BED4-B0EFDF6C7834
Core Storage disk: disk6
Finished CoreStorage operation


11a. Check the CoreStorage setup:

#diskutil cs list
CoreStorage logical volume groups (3 found)
|

+-- Logical Volume Group 71377E10-7126-4E7B-A52D-F96F383F56BA
    =========================================================
    Name:         FusionDrive
    Status:       Online
    Size:         1126902611968 B (1.1 TB)
    Free Space:   73728 B (73.7 KB)
    |
    +-< Physical Volume 4512A812-D998-4A45-AF5E-CB6F8EE4BD2D
    |   ----------------------------------------------------
    |   Index:    0
    |   Disk:     disk0s2
    |   Status:   Online
    |   Size:     127691702272 B (127.7 GB)
    |
    +-< Physical Volume 2344BA4E-2460-4631-B52A-BFFB6DBBA9C7
    |   ----------------------------------------------------
    |   Index:    1
    |   Disk:     disk1s2
    |   Status:   Online
    |   Size:     999210909696 B (999.2 GB)
    |
    +-> Logical Volume Family 6994CC0B-958C-4CF1-A4BF-7B7553478619
        ----------------------------------------------------------
        Encryption Status:       Unlocked
        Encryption Type:         None
        Conversion Status:       NoConversion
        Conversion Direction:    -none-
        Has Encrypted Extents:   No
        Fully Secure:            No
        Passphrase Required:     No
        |
        +-> Logical Volume 04085D2E-D630-4EED-BED4-B0EFDF6C7834
            ---------------------------------------------------
            Disk:               disk6
            Status:             Online
            Size (Total):       1118375247872 B (1.1 TB)
            Size (Converted):   -none-
            Revertible:         No
            LV Name:            Macintosh HD
            Volume Name:        Macintosh HD
            Content Hint:       Apple_HFS

12. Start CarbonCopyCloner and clone back the USB boot drive to your newly created FusionDrive (named "Macintosh HD" in step 11).
13. Reboot your system and try to boot the recovery partition by pressing the ALT key during reboot.
14. Reboot your system from the FusionDrive.
15. Enable Trim support, to keep your SSD speed high over time. You can patch the OSX driver yourself, or you use tools like http://www.groths.org/trim-enabler/ or Chameleon Trim Enabler.



For reference, here is the partition printout of a stock Apple 2012 iMac with original FusionDrive configuration:
# diskutil list
/dev/disk0
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      GUID_partition_scheme                        *121.3 GB   disk0
   1:                        EFI                         209.7 MB   disk0s1
   2:          Apple_CoreStorage                         121.0 GB   disk0s2
   3:                 Apple_Boot Boot OS X               134.2 MB   disk0s3
/dev/disk1
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      GUID_partition_scheme                        *1.0 TB     disk1
   1:                        EFI                         209.7 MB   disk1s1
   2:          Apple_CoreStorage                         999.3 GB   disk1s2
   3:                 Apple_Boot Recovery HD             650.0 MB   disk1s3
/dev/disk2
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:                  Apple_HFS Macintosh HD           *1.1 TB     disk2


Sunday, February 3, 2013

Android "Framework" a big mess

I think every serious developer creating apps for IOS and Android was bitten by one of the countless bugs in the Android framework and / or device bugs due to fragmentation.

It seems Google is not taking this problems serious enough. Over 28.000 open bugs in the Android Bugtracker. Many of them were created years ago and are real show stoppers, but Google is simply not fixing them. It seems to me they fix bugs only if they need the feature for one of the Nexus devices.

For example the HTTP Live Streaming bug. Opened in May 2011, it is still not fixed. But Google recently set the status of this bug to "Spam". Spam? Tsss.