Tips and best practices for migrating a legacy website into Drupal 7 with the Migrate API

drupal_articleLast year, we have migrated a site from a legacy CMS with a lot of custom code into Drupal 7. With the Migrate module (which is now in Drupal 8 core by the way), it was not really hard to do, even though the source database was not really well-designed. The site had around 10k users and 100k entries which ended up being nodes in Drupal, with around 700 videos to migrate. While this is still not a huge migration by any means, it was big enough to make us think about the best practices for Drupal migrations.

We have collected some tips and tricks if you need to do anything similar:

Building the right environment for the migration work

If you would like to work on a migration efficiently, you will need to plan a bit. There are some steps which can save you a lot of time down the road:

  • As always, a good IDE can help a lot. At NewPush, we are using PhpStorm – in that, keeping both the migration code, the source and the target databases opened is straightforward. Also, it is really easy to filter the database table displays looking for a given value which comes really handy. Anyway, any tool that you can use is fine, as long as you are able to quickly check the difference between the source and target versions of the data, which is essential.
  • I know this can be a bit of a pain sometimes, but still: try to ensure that the site can be built from scratch if necessary. This is where techniques like Features step in. I will not explain this in detail since it is outside the scope of the article. Just make sure you ask yourself the question: “what if I really mess up the configuration of this site?” (In a real world scenario you will often need to adjust things like content types, field settings etc. etc. and you will probably need a method for keeping the final configuration in a safe place.)
  • Before getting started, try to figure out a good way for validating the migration. This is kind of easy if you have 20-30 items to move over, but when you need to deal with 50k+ nodes, it is not going to be that trivial – especially if the source data is not very clean.

How to work with your migrations

The next thing to work on is optimizing your day-to-day work. Making sure that you can perform the basic tasks fast is essential, and you will need to figure out the best methods for testing and avoiding errors as well.

  • Use Drush as much as possible. It is faster and less error-prone than UI. There are a lot of handy parameters for migration work. Talking about the parameters, the –feedback and the –limit switches are really handy for quick testing. With the –idlist parameter, you can exactly specify what to import, which is great for checking edge cases.
  • Try to run the full migration from time to time. This is usually not very convenient in a development environment, so having access to another box where you can leave a migration running for hours can make things quite a bit easier.
  • Don’t forget to roll back a migration before modifying the migration code – it is better to avoid database inconsistency issues.

Migration coding tips

Talking about coding the migration itself, there are several things to consider, most of them are pretty basic coding best practices:

  • Try to extract the common functionality into a parent class. You will probably need some cleanup/convert routines; this is the best place for them.
  • Try to document your assumptions. Especially when dealing with not-so-clean data this can be really useful. (Will you remember why the row with ID 2182763 should be excluded from a given migration? I will not, so I always try to add a bunch of comments whenever I find such an edge case.)
  • Use the dd() function provided by Devel – this can make things easier to follow.
  • You will most likely run into the “Class MigrateSomething no longer exists” error at some time. Give it a drush migrate-deregister –orphans and it will go away.

How to optimize the performance of your migration code

It is no surprise that running a full migration often takes a very long time.

  • Other modules can greatly affect the speed of migration/rollback. Try to migrate with only a minimal set of modules to speed things up. This is the easiest way to get some extra speed. (And also if you are on a dev machine, make sure that Xdebug is not active. That is a “great” way to make things much slower.)
  • Using the migrate_instrument_*() functionality, you can figure out the bottlenecks in your migration.
  • In hook_migrate_api(), you can temporarily disable some hooks to speed up the migration.

I hope you have found some useful tips in this article. If you have a question, a tip to add, or perhaps an interesting story about a migration that you have done, just share them in the comments, we are all ears!


Domino 8.5 SELECT Failure, Cannot Select Mailbox: Entry Not Found in Index

Problem opening IMAP INBOX with Domino 8.5

If you are getting an error

Mail was unable to open this mailbox on the server “domino-server.yourdomain.com”.

The server returned the error: SELECT failure, cannot select mailbox: Entry not found in index

Most likely the IMAP INBOX for that user has been corrupted. The solution is simple and quick even on large mailboxes

  • Disable IMAP NSF support for user’s mailfile by issuing the command: load convert -e- …mail/usermail.nsf
  • Enable IMAP NSF support for user’s mailfile by issuing the command: load convert -e ..mail/usermail.nsf

Upgrading Android on Samsung Galaxy Pro B7510

The official tool to upgrade the Samsung Galaxy Pro B7510 is Kies, but in most cases, the upgrade can’t be done. Instead there is a tool named Odin:
http://enzag.com/technology/android/how-to-upgrade-samsung-galaxy-pro-b7510-to-gingerbread/

It is recommended to register at SamFirmware where you can download the official firmware, then you can install it using:

http://www.sammobile.com/2014/12/23/firmware-for-samsungs-upcoming-tizen-phone-the-z1-now-online/8/?view=2588

 


Cognos 10.1 install on CentOS 6.3 64 bit

  1. yum update (then reboot if kernel has been patched)
  2. yum install glibc.i686
  3. yum install openmotif
  4. yum install libgcc.i686
  5. yum install openmotif22
  6. yum install openmotif22.i686
  7. yum install xauth
  8. yum install libXtst
  9. tar xvzf bisrvr_linuxi8664h_10.1.1_ml.tar.gz
  10. cd linuxi38664h/
  11. ./isetup

How to Flatten a Dimensional Data Model Built in Excel with Pivot Tables

To flatten a dimensional model built in Excel using pivot tables, follow these steps:

  1. save the original worksheet with the pivot table as a CSV (tab delimited is even better)
  2. select the range or column that contains the blanks (if you have blanks under the header, don’t select the column header in the range)
  3. select Edit -> Go to… -> Special
  4. select blanks (at this point Excel will select all the blank cells
  5. press equal
  6. point to the cell above the first selected cell
  7. press <ctrl>+Enter and Excel will copy the formula to all the blank cells
  8. to replace the formula by the values, simply save the worksheet again as CVS, or copy and paste special the cells as values
  9. that’s it, you’re done. (Don’t forget to save.)

CSF / LDF user / IP lock out info

Where can we view the lock out triggers / logs?

There are a few ways to do it.

The log itself is in /var/log/lfd.log – this provides you with all the information about what lfd is doing. lfd is the process that keeps track of many things: login failures (technically it is called the “login failure daemon”) but also other irregularities on a hosting platform: long running processes, user-run script executions, root logins, things like that.

Another way is to look at the output of the ‘csf -g’ command.
Its full use is: csf -g ip.address.goes.here

This will show you real-time whether or not a certain ip is accepted or dropped (denied). If the IP does not show up when searched this way, then csf/lfd have no blocks or accepts on the IP in particular, at which point server-wide firewall settings will still apply; for example a server-wide deny to a certain port.

To manually unblock, you can use the ‘csf -dr ip.addr.here

More usage information on csf is in the output of the ‘csf‘ command, as well as the author’s website at http://configserver.com/cp/csf.html .


Setting up FileZilla for connecting to the server (first time only)

 

  1. Start FileZilla. I’m using FileZilla 3.5.0 for this tutorial, but the steps should be the same in any recent version.
  2. We will start by creating a new site in FileZilla so that we won’t have to fill the credentials every time we would like to connect to the server. Open File > Site Manager… and click on the New Site button on the left. Edit the name of the site so it’s easy to recognize.
  3. Fill in the form on the right with the following information:
    • Host: [hostname/IP addess]
    • Port: [FTP port]
    • Protocol: FTP File Transfer Protocol
    • Encryption: Require explicit FTP over TLS
    • Logon type: Account
    • User: [your FTP username]
    • Password: [your FTP password]
    • Account: [the domain name for the user in Windows Server] Click on Connect to test the FTP connection.
  4. An Unknown certificate window will pop up. This means that FileZilla recognized that we are using a secure connection and asks whether the information in the certificate is legit. If it is, then click on *OK-. (You might want to check “Always trust certificate in future sessions” so that FileZila won’t ask this again.)
  5. Wait a little until the server processes the login request – it shouldn’t take more than 10 secs on a decent connection. If everything went right, the folders/files will appear on the right pane of FileZilla.

After this, you can use the FTP software as usual for downloading/uploading files to the server.

Connecting to the server

The next time you would like to access the FTP server just…

  1. Start FileZilla.
  2. Open up the Site Manager from the File menu.
  3. Select the saved site from the left and click on Connect.

FileZilla already knows the details of the connection from the steps above, so it should log in without any problem.


WHMCS OpenSRS sync error for a domain

Problem: OpenSRS domain sync error

Rarely for a domain under OpenSRS management, the following error show up for each operation: “Details could not be retrieved for your domain.com. Error: Check your browser's encoding type, and modify it to reflect your password's encoding type. For support regarding yourdomain.com, please contact your reseller

Solution: the OpenSRS domain password

Initially we looked at the troubleshooting guide, but couldn’t find the relevant info. The WHMCS support folks pointed us in the right direction:

This is a very rare error, it is caused by a character in the domain’s password field in the mod_opensrs table that OpenSRS’ API does not recognise.
To resolve the issue, simply change the domain’s password via the OpenSRS control panel to that does not contain special characters, then using a tool such as phpmyadmin or RazorSQL edit the domain’s record in the mod_opensrs table of the WHMCS database and enter the new password directly.
As always before making changes to the database please backup.


Skunk Works

Skunk Works

 

<<The nickname Lockheed employees gave to their organization is an adaptation of the name for the ultra-secret high security “Skunk Works” moonshine operation featured in Al Capp’s widely popular comic strip Li’l Amber, which ran from 1934 to 1977.>>

— Smithsonian Air and Space Museum

The name since became widely used for elite research teams especially in the hi tech industry.

Panel from the Smithsonian
Quote from the Air and Space Museum

How to uninstall IBM Lotus Notes client from the Macintosh on OS X

You can uninstall IBM® Lotus® Notes® by dragging Notes.app from /Applications to the trash. This preserves user data.

You can also uninstall Notes using the uninstaller application supplied with the Notes install media. This preserves user data as well.

You can also uninstall Notes by dragging the following items to the Apple® Mac OS X® trash bin:

  • Notes.app
  • ~/Library/Application Support/Lotus Notes Data folder (“~” = user’s home directory)
  • ~/Library/Preferences/Notes Preferences
  • /Library/Receipts/Lotus Notes Installer.pkg
  • /Library/Receipts/Lotus Notes Installer_English.pkg
  • /Library/Receipts/xpdcoreinstaller.pkg

via IBM How to uninstall Notes client from the Macintosh – United States.