Kernel panic with 3.2.0-43

May

31

2013

I recently had a problem with a server throwing random kernel panics. I’ll just post the problem and solution here in case anybody else finds it useful.

After about 12-20 hours of heavy system load, the backup server (Ubuntu Server, Precise 12.04.2 LTS) would freeze with a kernel panic. The problem was very reproducible, and would always happen after long periods of heavy load. The system was completely unresponsive, and even SysRQ failed (although this possibly had to do with the system not detecting a new keyboard after the panic).

This was the panic (transcribed):

Kernel panic - not syncing: Attempted to kill init!
Pid: 1, comm: init Not tainted 3.2.0-43-generic #68-Ubuntu
Call Trace:
[<ffffffff81645fb1>] panic+0x91/0x1a4
[<ffffffff8106bf55>] forget_original_parent+0x245/0x250
[<ffffffff8106bf77>] exit_notify+0x17/0x110
[<ffffffff8106c803>] do_exit+0x1f3/0x450
[<ffffffff8106cc04>] do_group_exit+0x44/0xa0
[<ffffffff8107de3c>] get_signal_to_deliver+0x21c/0x420
[<ffffffff81014865>] do_signal+0x45/0x130
[<ffffffff81130000>] ? shmem_fill_super+0x190/0x1f0
[<ffffffff8101ecf6>] ? save_i387_xstate+0xe6/0x230
[<ffffffff81014b15>] do_notify_resume+0x65/0x80
[<ffffffff8165f07c>] retint_signal+0x48/0x8c
panic occurred, switching back to text console

The panics started to happen after a system upgrade, so it appeared to be more of a software than hardware problem. We decided to test the memory anyway, but the test was negative even though it memtest86 performed 26 passes.

I was considering reporting this as a bug to the Ubuntu bug tracker, but the next day a new kernel was submitted to the repositories (3.2.0-44). After performing the upgrade, the system has gone four days of heavy load without a hiccup and I consider this problem solved.

Master Thesis published

Nov

6

2012

My Master Thesis has now been published and can be found at the LiU Electronic Press archive. I wrote this thesis at my time at ABB Corporate Research in Västerås where I performed research into electric breakdown in transformer oil.

J. Fors, Modeling and OpenFOAM simulation of streamers in transformer oil. Master thesis. Linköping University, 2012. [pdf] [ref]

Electric breakdown in power transformers is preceded by pre-breakdown events such as streamers. The understanding of these phenomena is important in order to optimize liquid insulation systems. Earlier works have derived a model that describes streamers in transformer oil and utilized a finite element method to produce numerical solutions. This research investigates the consequences of changing the numerical method to a finite volume-based solver implemented in OpenFOAM. Using a standardized needle-sphere geometry, a number of oil and voltage combinations were simulated and the results are for the most part similar to those produced by the previous method. In cases with differing results the change is attributed to the more stable numerical performance of the OpenFOAM solver. A proof of concept for the extension of the simulation from a two-dimensional axial symmetry to three dimensions is also presented.

omskakas.se version 3.0

Nov

5

2012

As time goes by, things change. This site, omskakas.se saw the light six years ago in September 2005 when I still was a high school student. The old WordPress installation has done a good job, going from version 1.5.2 to version 3.4.2 with the same database. That’s about 60 incremental upgrades, some of which have contained major changes. The fact that the old blog still functioned perfectly before being archived is in my opinion a good testament to the reliability of WordPress.

The whole site has been rebooted and the old content is gone. I’m keeping the address, but you’ll see a new theme, fresh content and hopefully I’ll be able to write some interesting posts in the future.

The old blog did have some posts that had generated significant traffic and still received a small trickle of visitors. All of these posts (except one) are now highly outdated and I am confident that they won’t be missed now that they’re gone. The sole exception is the information I provided about the Thinkpad T30 memory slot failure which generated many pages of valuable insight from visitors all around the world. As of October 2012, those posts represented more than 25% of all visits to this site. I’ll make sure that information stays online, stay posted about that.

All in all, a fresh start for omskakas.se. Version 3.0.

Jonathan

 

Memory slot updates 7

Aug

19

2007

Update, November 2012: This is an archived blog post and has been salvaged from an earlier instance of this site. The contents below is outdated (August 2007). Read on your own risk.

I have recieved a few e-mails about the progress since my last post about the Thinkpad T30 memory issues. I went to the electronics workshop to check up what they would charge for resoldering the motherboard slots. Since they have hourly rates around 300 SEK/hour (~32€/hour) the price would land on either 300 or 700 SEK, but they wouldn’t give any warranty for assuring the machine would work afterwards. I felt the price a bit too expensive and didn’t want to risk this machine, not yet at least.

I went home and waited, and finally the new memory slot cover that Lenovo sent me arrived. The new cover is much more rigid than the old one, and has like a square emboss over it. This allowed me to put a few pieces of paper under it to apply pressure on the slots, making them work. This worked for a few days and then it didn’t again. Probably the solderings get worse when I try to do this, so I have stopped applying pressure to the slots. Now there seems to be no solution to the problem of the broken memory slot than to do a real re-soldering.

One finnish guy, Hobbymat, responded to the last post and wrote the following about re-soldering the Thinkpad:

Greetings from Finland! I have repaired a few T20/T30 series Thinkpads using a very fine-tipped soldering iron (a Metcal, a Weller will also do) and a stereo microscope. In my opinion the memory slot repair procedure does and should not need application of any extra solder as mixing different solder metals (conventional tin-lead mixture used in older Thinkpads or lead-free tin-silver-copper solder used in newer Thinkpads) is not recommended, and it is rather difficult to tell which solder has originally been used. Actually the needed amount of solder metal is already there in the joints – the only problem is that there are cracks that do not conduct electricity any more.

However, it is advisable to use liquid soldering flux applied to the solder joints with a small brush or a “flux pen” prior to re-heating and melting of the individual solder joints one by one. The flux penetrates the cracks in the old solder joints and it will help to remove the oxide layer in the crack when the existing solder melts, producing a good-as-new solder joint.

As spotting all the broken memory module slot joints is by far not easy even for a professional with the best available equipment, it’s best to fix all the solder joints if you decide to go that way. There are as many solder joints to “flux and re-solder” as there are pins in a memory module. That is, 144 joints for a T21 or 200 joints for a T30, and you’ll need to double this number if you are going to use a memory module in both slots! But it can be done, given some time and patience.

This repair procedure is suitable for a skilled electronics hobbyist with access to the tools mentioned above. If you don’t have a stereo microscope, you can also use a jeweler’s loupe, good loupe glasses or a very good magnifier for viewing the work area. However, most likely you can’t spot ANY of the micro-fractures of the solder joints unless you have a decent stereo microscope with a magnification between 10X and 40X. Of course, a very steady hand is also needed for this repair job!

Hope this helps someone :)

All I need to find now is a skilled solderer that can do this for cheap. Since I now live in Linköping it should not be too difficult to find somebody at the Institute of Technology at which I now am studying.

Please comment if you have ideas or other things to add to the discussion about the Thinkpad T30 memory slot failure.

Thinkpad T30 memory slot failure 51

Jul

13

2007

Update, November 2012: This is an archived blog post and has been salvaged from an earlier instance of this site. The contents below is outdated (July 2007). Read on your own risk.

Update: Read the followup here

My Thinkpad that I bought used a year ago has the dreaded memory slot failure that results from a design error in the memory slot design found in almost every T30 manufactured. What happens is that after some time of usage and heat generation one of the memory slots cease to function. Since the T30 only has two slots from the beginning, this severely restricts the amount of memory that can be installed now. 512 MB of Ram is getting a little to slim because I heavily use VMWare on my computer, running one guest operating system as well as my Linux host system.

IBM did replace these faulty motherboards for a time, warranty or not, but that service ceased in the middle of 2005. Since the warranty on the machine expired in May 2006 and I bought it in August, I’m a bit annoyed of having to live with this problem. The previous owner apparently did not know about the free replacement if still under warranty.

After having spoken to a lot of support technicians on IBM/Lenovo the last week there is apparently no chance whatsoever of me getting a free/cheap motherboard replacement. I have tried to use stacked paper under the memory cover, but this does not work well enough. I finally called the Swedish spare parts office at Lenovo to find out what it would cost to replace the motherboard, just for the fun of it.

Luckily, the guy on the spare parts division was just the person I needed to talk to. This was an experienced, friendly technician who works more with the actual machines more than talking to the customers. He really knew about the problem with the model and confirmed what I had observed. He explained the problem to me:

  • First, there is to little space between the memoy cover and the memory modules, not giving the modules enough space to cool
  • The small space also means that having the laptop in your lap puts pressure on the memory modules themselves
  • There is a minor design error in the hard drive sled which is located just above the memory that also contributes to overheating
  • There is a blue plastic shock damper deep inside the computer that puts too much pressure on the memory modules from underneath, eventually loosening the solders between the memory slot and the motherboard.

He told me that what I could do is to get the machine to an experienced solderer to solder up the connectors on the motherboard. He pointed out the group of connectors that is affected. The problem can be seen if looked through a magnifying glass and pushing on the memory slot. Let the solderer put on quite a lot of solder, preferably of the lead-based type and not the silver-based one since lead makes the solder more soft and flexible. He also sent me the new type of hard disk sled and a new design memory cover for free!

This solution will last some time, but the main culprit is the blue plastic shock damper. Even with a re-soldered motherboard it will just be a matter of time until the solders get loose again. What really should be done is to take apart the whole machine to get deep inside, remove the plastic shock and put it all back again. He told me it would be a big job to do, and since the machine probably will last another two years without it. Still, I’m not sure what to do here.

For all of you out there who have a malfunctioning T30, there is still hope outside the warranty. If your warranty is out; Sorry, no free replacement form the support team. Document MIGR-44772 does no longer exist on the Lenovo servers since they apparantly have removed it. (MIGR-44772 was an internal IBM/Lenovo document describing that the T30 owner shoud get a free repair).

Most of all I am happy that I was able to speak to a guy at the spare parts division that really knows what he is talking about, and that he sent me these parts for free. He even gave me the address to a local shop here in Västerås that is really good at doing these soldering jobs for a low price.

This afternoon I am going to the shop to check out what they can do. Right after that I am buying myself a new battery that my job pays for. Finally being able to work unplugged! Wee!

Your own internet radio station with MPD/Icecast 16

Jun

20

2006

Update, November 2012: This is an archived blog post and has been salvaged from an earlier instance of this site. The contents below is outdated (June 2006). Read on your own risk.

This is a short, un-detailed howto of how to create your own internet station using the MPD music player daemon and icecast. There are some drawbacks with this solution, but since I have been an avid MPD user for a long time, this solution integrates nicely with what I am used to. Read on for complete instructions
Advantages:

  • Integrates directly with MPD
  • Quite customizable and very nice to control via an SSH shell.

Drawbacks:

  • Ogg Vorbis is the only audio format supported as of now
  • There is a bug when resuming playback that has been paused for a long time

Prerequisites

  • Your favorite Linux distribution. On this current server I have gentoo, while my favorite is Ubuntu. Your choice!
  • A good internet connection. Streaming music over a slow internet connection will probably halt your other traffic.

Software used:

  • mpd, at least version 0.12.0
  • icecast, I use version 2.3.1
  • your favorite MPD client, I favour “ncmpc”
  • a listener client, on Windows I use foobar2000 (you need ogg support for your player)

Let’s get started. Install all the above software on your computers, see the howto on the wiki for this.

Configure icecast. This can be a bit tricky and ther are many ways to fail in this step. You should:

  • Set hostname (mine is stream.broach.se)
  • Set the source password (the password that mpd will have to specify to be able to stream to the icecast server)
  • Set the admin password.

Icecast can be a bit tricky to configure, especially since it uses an XML file. The configuration is probably located at /etc/icecast/icecast.xml.
You also need to choose whether to use burst-on-connect. I have turned burst-on-connect on to reduce latency between MPD and the listener client, and I have no problems with lag. Remember that I mostly sit on high-speed internet connections and therefore your situation and configuration might vary.

Now configure mpd. See the howto on the wiki again. I use quality 4.00, which gives a bitrate of a bit lower than 128, which is OK for web-radio. If you are an audiophile you should use 5.00 or more. Also set the correct password needed to connect to the icecast server. This is the source password you have typed in the icecast xml file. If everything seems OK, fire up icecast and then MPD.

My mpd.conf:
[sourcecode language="css"]music_directory “/var/music”
playlist_directory “/var/lib/mpd/playlists”
db_file “/var/lib/mpd/database”
log_file “/var/log/mpd.log”
error_file “/var/log/mpd.error.log”
pid_file “/var/run/mpd/mpd.pid”

audio_output {
type “shout”
name “Music stream”
host “localhost”
port “8000″
mount “/stream.ogg”
password “password”
quality “2″

user “source”
description “All kinds of music”
genre “Everything”
} # end of audio_output[/sourcecode]
Replace “user” and “password” with the information you specified in your icecast configuration.
Start icecast, then mpd, and then your mpd client to play some music. Now mpd streams music encoded as an Ogg Vorbis stream to icecast, which is now waiting for listeners. Goto another computer and start to listen to “http://your-server-adress:8000/stream.ogg.m3u”, which will be the name of the audio stream.

Everything should work now. If it doesn’t, you probably have a configuration issue. If the playback is skippy and it seems that MPD plays the file at wrong speed (ie. the timer seems to go very fast and jump a few seconds now and then), you have encountered the skipper bug. Restart MPD and resume playback. I don’t have another solution to this bug yet, but luckily it only occurs when you haven’t played any music for a few minutes. Prevent it to happen by always have a song in your MPD playlist. This bug was solved by upgrading libshout from 2.0 to 2.1. Thanks to the guys in the #mpd IRC channel for this!

Wishlist:

  1. A configuration option for MPD that sends a silent audio stream when the music is offline or paused. This prevents the skipper bug, and allows all clients to be connected even though you pause the music. You won’t have to reconnect again when starting to play, because there always has been a connection
  2. Reverse proxy support for icecast. I have the icecast server located behind my router, and the generated URL is invalid because port 80 on the webserver is routed to the icecast server for the domain “stream.broach.se”, but not “etnoy.broach.se” and “www.broach.se”. Therefore I need to manually change the source code to make this work. I resolved this problem by manually adding a stream.ogg.m3u file to the web directory containing the correct URL for the strem. Hackish, but it works well.

Report any problems by commenting on this blog entry, or ask me a question by e-mail.

Updated 06/07 2006
Updated 11/01 2006
Updated 11/03 2006
Updated 07/01 2007