Cannot drop secondary log file

SQL server 2012 (not sp1) 64 bit ent.

The database is in simple recovery. During a large transaction I added a secondary log file not to let it fail.

Now I made the primary log file 50gb in size, secondary 1mb in size with no autogrow. 

tried

Checkpoint, shrink, remove file -- no luck

insert 2 million rows into a table to create some log data and then Checkpoint, shrink, Checkpoint, remove file -- no luck

detach database , attach_with_rebuild log -- no luck

I cannot seem to get rid of this log file, any ideas?

DBCC loginfo

RecoveryUnitId FileId      FileSize             StartOffset          FSeqNo      Status      Parity CreateLSN
-------------- ----------- -------------------- -------------------- ----------- ----------- ------ ---------------------------------------
0              2           54470246400          8192                 36          2          128    0
0              9           253952               8192                 35          0          64     0

I get the below error every time

The file 'XXXX' cannot be removed because it is not empty.

---

Gokhan Varol