blog

Another btrfs test for our squid servers

Tue, 08 May 2012 17:00:59 +0200

After almost 2 years I decided it was time to try btrfs yet another time on our squid proxies.

And, to my great amazement, this time btrfs did not fail. It "just worked". Yet I saw something strange: The machine would encounter "waves" of high load, probably induced due to periods of high IO wait. The IO activity itself didn't show any "wavy" patterns, though.

A major fuckup on part of spamhaus!

Fri, 04 May 2012 14:37:29 +0200

A major fuckup on part of spamhaus: http://www.spamhaus.org/sbl/query/SBL138067

The evidence section listed "inetnum: 95.218.0.0 - 95.219.255.255", yet spamhaus listed 93.218.0.0/15 (first octet 93 instead of 95)!

The error has been rectified within 20 minutes of my report, though!

X-Forwarded-For: unknown considered harmful

Fri, 04 May 2012 14:36:06 +0200

Recently a colleague and I wanted to submit a few FP to the ClamAV project. So we went to http://cgi.clamav.net/sendfp.cgi just to find an "Under Maintenance"-page. So we tried a few days later, just to find the same page.

So I asked on the clamav-users mailinglist and found out that the page was in fact working. Just not for us. Oh well. So I asked their admin to look into their varnish logs. He wouldn't do that.

Today, I dug deeper and found this:

Their server doesn't like the "X-Forwarded-For: unknown" header!
(see http://www.squid-cache.org/Doc/config/forwarded_for/)

On our squids it was set to:
forwarded_for off
which results in a
X-Forwarded-For: unknown
header - and a subsequent error page from varnish. Setting it to "delete", "on" or "truncate" makes the page http://cgi.clamav.net/sendfp.cgi work
again.

Only "off" causes the page to fail.

AOL, still idiots after all these years

Thu, 03 May 2012 13:07:00 +0200

Today, we quite a lot of mail piling up for AOL, because of this:

May 3 12:53:10 mail2 postfix/smtp[29232]: 3VjsWg426Lz1tTY: host mailin-04.mx.aol.com[64.12.90.34] refused to talk to me: 554 mtain-mh05.r1000.mx.aol.com ESMTP not accepting connections

So I ventured to http://postmaster.aol.com and submitted a ticket.
Well, at least I tried.
Filled in all the data and then failed to solve the first captcha. Once I solved another captcha, I got the message that my ticket would not be accepted because I already submitted another ticket withtin 24 hours. WTF?

CISCO breaks DKIM on their ASA/PIX (again)

Thu, 08 Dec 2011 09:49:54 +0100

site which was using Postfix and a CISCO ASA with "smtp protocol

fixup" enabled.

I was able to work around the delivery problems by stripping the DKIM

headers on outgoing mails (as so often).

Some interesting info got out:

I've also discussed these results with local Cisco support and they

confirmed it's a known bug (not published) with DKIM and smtp inspection

engine in latest IOS versions.

This should be fixed in some newer IOS version (8.4(10)) which is not

public yet (latest is 8.4(2)).

Over the last few days I discussed SMTP delivery problems with a czech site which was using Postfix and a CISCO ASA with "smtp protocol fixup" enabled.

I was able to work around the delivery problems by stripping the DKIM headers on outgoing mails (as so often).

Some interesting info got out:

I've also discussed these results with local Cisco support and they confirmed it's a known bug (not published) with DKIM and smtp inspection engine in latest IOS versions.

This should be fixed in some newer IOS version (8.4(10)) which is not public yet (latest is 8.4(2)).

Working around broken CISCO/PIX or ASA installations

Mon, 20 Jun 2011 16:23:13 +0200

CISCOS's firewalls, predominantly the PIX and ASA have a feature called "smtp protocol fixup".

There's nothing to fix, yet some admins enable this feature, not knowing that it degrades performance and blocks legitimate mail - because the SMTP engine in the firewall doesn't correctly parse DKIM-Signature headers!

Great job, Cisco!

A workaround is to strip the DKIM-Signatur eheaders when sending to sites which expose this behaviour:

Jun 20 15:37:04 mail postfix/smtp[29987]: 3QyXw43YY0zFvnF:
 to=<v.marmol@skynet.be>, relay=in.mx.skynet.be[195.238.5.129]:25,
delay=1571, delays=1214/0.04/0.08/357, dsn=4.4.2, status=deferred (lost connection with in.mx.skynet.be[195.238.5.129] while sending end of data -- message may be sent more than once)
Jun 20 15:37:04 mail postfix/smtp[29987]: 3QyXw43YY0zFvnF:
 to=<...@skynet.be>, relay=in.mx.skynet.be[195.238.5.129]:25, 
delay=1571, delays=1214/0.04/0.08/357, dsn=4.4.2, status=deferred (lost connection with in.mx.skynet.be[195.238.5.129] while sending end of data -- message may be sent more than once)

In transport_maps I defined:

skynet.be               nodkim:

and in master.cf I defined a new transport:

nodkim    unix  -       -       -       -       -       smtp 
     -o smtp_header_checks=pcre:/etc/postfix/no_dkim.pcre

no_dkim.pcre contains the pattern used to identify the offending header:

/^DKIM-Signature:/  IGNORE

New fail2ban rules for postscreen

Thu, 02 Jun 2011 09:48:05 +0200

fail2ban doesn't work properly with the new postscreen log entries, since those are different from the ones smtpd is generating.

postscreen:

#Jun 2 09:36:15 mail postfix/postscreen[14514]: NOQUEUE: reject: RCPT from [27.74.129.153]:11010: 550 5.7.1 Service unavailable; client [27.74.129.153] blo$

#Jun 2 09:37:28 mail postfix/smtpd[14544]: NOQUEUE: reject: RCPT from smtpgv01.qualitysmtp.com.br[187.85.160.22]: 550 5.1.1 <anna.hegele@charite.de>: Recip$

Jun 2 09:36:15 mail postfix/postscreen[14514]: NOQUEUE: reject: RCPT from [27.74.129.153]:11010: 550 5.7.1 Service unavailable; client [27.74.129.153] blo$

smtpd:

Jun 2 09:37:28 mail postfix/smtpd[14544]: NOQUEUE: reject: RCPT from smtpgv01.qualitysmtp.com.br[187.85.160.22]: 550 5.1.1 <anna.hegele@charite.de>: Recip$

So I changed the regexp to read:

failregex = reject: RCPT from (.*)\[<HOST>\]:([0-9]{4,5}:)? 550

I made the :portnumber stuff optional and alas, it's working now!

Nokia Deppen II

Mon, 30 May 2011 17:22:41 +0200

Heute kam Peter mit seinem Handy rum, und wollte Eduroam konfiguriert haben.

Stellt sich raus, daß die Settings im Netzwerkdialog falsch übersetzt sind! Ciphers sind "Ziffern" (statt "Chiffren") und "Gebiet" ist in Wirklichkeit "Domain". Oh Mann!

Handyvertrag

Mon, 30 May 2011 09:35:29 +0200

Ich habe gestern eine Analyse von Constanzes Handyverhalten gemacht und festgestellt, daß wir mit einem Wechsel des Anbieters von 40,- pro Monat auf 10,- Monat fallen UND eine Internet-Flat dazukriegen. Die, Congstar, die!

smartmobil.de for teh win!

Nokia Deppen

Sun, 29 May 2011 22:07:59 +0200

Der Hirnboy-Preis der Woche geht an: Nokia, für Nokia Maps. Bzw. die geniale Idee, eine Map via WLAN herunterladen zu können, um sie später offline nutzen zu können.

Tolle Sache: Funktioniert nur nicht.

Stellt sich heraus, daß Ovi Maps UNBEDINGT für ein Update die "Internet" Zugangsklasse nutzen will, auch wenn man ihm gesagt hat, es möge bitte nur die Klasse "Free" nutzen. Fehlermeldung: Total unklar. Ein Forumspost hat mich darauf hingewiesen, dass das benutzte WLAN in der "Internet" Zugangsklasse sein muss.

LinuxTag 2011

Sat, 14 May 2011 11:28:41 +0200

Gestern war ich auf dem Linuxtag 2011 um meinen Vortrag über die Neuerungen in Postfix 2.8 zu halten. Hinterher wurde ich von einem inovex Mitarbeiter auf deren Stand eingeladen; dort hatten wir einen kurzen Blick auf die Postfix Installation eines großen Kunden geworfen. Der hat ein Performanceproblem welches voraussichtlich durch Postfix 2.8 zu beheben sein dürfte.

Dort allerdings habe ich bemerkt, daß bei uns in der Nacht ein Spammer ein gephishtes Passwort eines unserer hochintelligenten User benutzt hat um 250.000 Spammails zu verschicken. Die Aufräumarbeiten dazu dauerten dann auch 30 Minuten, die ich dankenswerterweise am inovex Stand verbringen durfte.

Hinterher gings dann weiter zu OTRS wo ich Martin Edenhofer (der Erfinder) und Alexander Halle (OTRS Community Board) kennenlernen durfte.

Später trudelten dann Marc Schiffbauer und Martin Dummer ein, mit denen ich CAcert Zertifizierungen gemacht habe und ein frugales Mahl in der Cafeteria genoß.

Android Gingerbread on a HTC Desire

Sat, 02 Apr 2011 11:30:02 +0200

I bought a HTC Desire and installed the most recent Oxygen release on it. This is a significant improvement over the look and feel of Symbian on my old Music Express 5800.

Also, the complete lack of shitty PC-based applications like "Ovi Suite" is a HUGE improvement.

CNAME and other data

Mon, 21 Mar 2011 13:07:25 +0100

# host test.update.microsoft.com

test.update.microsoft.com has address 80.156.86.78

test.update.microsoft.com has address 62.157.140.133

Host test.update.microsoft.com not found: 3(NXDOMAIN)

# host test.update.microsoft.com
test.update.microsoft.com has address 80.156.86.78
test.update.microsoft.com has address 62.157.140.133
Host test.update.microsoft.com not found: 3(NXDOMAIN)
Host test.update.microsoft.com not found: 3(NXDOMAIN)

Huh?

# host -t CNAME test.update.microsoft.com
test.update.microsoft.com is an alias for testupdate.microsoft.com.nsatc.net.
# host -t A test.update.microsoft.com
test.update.microsoft.com has address 80.156.86.78
test.update.microsoft.com has address 62.157.140.133

CNAME and other data?

OpenVPN 2 Cookbook

Wed, 09 Mar 2011 13:24:21 +0100

I spent the last year reading the OpenVPN 2 Cookbook chapters and reporting proposed changes back to the author.

And I must say it's a really good book. It's covering a lot of topics I had to explore on my own when setting up our OpenVPN installation at work, along with good examples and test cases.

Definitely the best book out there (so far).

Postfix on ZFS woes

Sat, 19 Feb 2011 10:43:00 +0100

See this bugreport by Mark Martinec:

http://www.freebsd.org/cgi/query-pr.cgi?pr=154873

Large packages on your system?

Tue, 08 Feb 2011 10:31:51 +0100

Which package is taking up the most space?

dpkg-query -Wf '${Installed-Size}\t${Package}\n' | sort -n

Microcode update?

Sun, 06 Feb 2011 18:59:44 +0100

Did you know that you could update the microcode on (some) Intel CPUs?

And did you know that Ubuntu/Debian come with an update utility as well as the most recent microcode files?

Neat!

sudo apt-get install intel-microcode microcode.ctl

This results (on an Acer Aspire One) in:

[    0.000000] Atom PSE erratum detected, BIOS microcode update recommended[    0.000000] Atom PSE erratum detected, BIOS microcode update recommended
[    0.004598] Atom PSE erratum detected, BIOS microcode update recommended
[    0.008000] Atom PSE erratum detected, BIOS microcode update recommended
[   18.455424] microcode: CPU0 sig=0x106c2, pf=0x4, revision=0x208
[   18.668142] microcode: CPU1 sig=0x106c2, pf=0x4, revision=0x208
[   18.673050] microcode: Microcode Update Driver: v2.00 <tigran@aivazian.fsnet.co.uk>, Peter Oruba
[   19.348014] microcode: CPU0 updated to revision 0x218, date = 2009-04-10
[   19.356912] microcode: CPU1 updated to revision 0x218, date = 2009-04-10

But keep in mind that

the kernel already worked around the bug
the microcode update is not permament (it doesn't survive a reboot, suspend or hibernate)
it survives a kexec reboot, though!

The only way is a proper BIOS update. Which - in my case - is not available.

Shame on you, Acer. Shame on you!

[ 0.004598] Atom PSE erratum detected, BIOS microcode update recommended

[ 0.008000] Atom PSE erratum detected, BIOS microcode update recommended

[ 18.455424] microcode: CPU0 sig=0x106c2, pf=0x4, revision=0x208

[ 18.668142] microcode: CPU1 sig=0x106c2, pf=0x4, revision=0x208

[ 18.673050] microcode: Microcode Update Driver: v2.00 <tigran@aivazian.fsnet.co.uk>, Peter Oruba

[ 19.348014] microcode: CPU0 updated to revision 0x218, date = 2009-04-10

[ 19.356912] microcode: CPU1 updated to revision 0x218, date = 2009-04-10

Acer Aspire One woes!

Wed, 02 Feb 2011 01:02:41 +0100

Today my old Acer Aspire One returned to me, because the internal microphone wouldn't work.

To cut a long story short: It was unclear why skype wouldn't use the micro or rather why the sound was really, really, low. But using PulseAudio I found out that disabling one channel of the micro caused the micro to work properly in skype. Yay!

ECC in Postfix

Wed, 26 Jan 2011 17:36:22 +0100

Postfix (when linked against OpenSSL 1.x) can use ECC (elliptic curve cryptography)

Starting with version 2.8.0, Postfix will automatically enable ECC.

If you want to enable perfect forward secrecy, you can enable ephemeral DH key exange using:

smtpd_tls_dh1024_param_file = /etc/postfix/dh_1024.pem

smtpd_tls_dh512_param_file = /etc/postfix/dh_512.pem

smtpd_tls_dh1024_param_file = /etc/postfix/dh_1024.pem
smtpd_tls_dh512_param_file = /etc/postfix/dh_512.pem

Generating these keys parameter files is easy:

openssl gendh -out /etc/postfix/dh_1024.pem -2 1024

openssl gendh -out /etc/postfix/dh_512.pem -2 512

smf-sav milter vs. postfix-2.8.x

Wed, 26 Jan 2011 15:53:52 +0100

Some people seem to use the smf-sav milter to implement sender address verification in Sendmail.

Unfortunately, this milter has several bugs:

it cannot handle multiline smtp banners
it cannot handle replies spanning over several packets
the author doesn't fix the bugs

Proof:

http://archives.neohapsis.com/archives/postfix/2010-09/0908.html