Uncategorized

SQL Server Parallel Index Defragmentation

  • by

I work for a company that has a pretty large SQL database. It’s about 4 terabytes in size. We had been using Ola Hallengren’s index maintenance script for a while but it got to be such that the script would take over 24 hours to run. That became a problem. That’s not to say there was really anything wrong with Ola’s script. It really is an excellent solution. However, it does only do index defragmentation serially … meaning one at a time. It works fine on smaller databases but when you have a HUGE database that no longer works well.

To solve this I created multiple jobs using his script and tried to break up the indexes between the jobs to have somewhat of a balance between them. That became a pretty big hassle to maintain. I would have to every-once-in-a-while go in and tweak it and some days it would work fine and other days it would not. So, I set out to see if anyone had created a solution to defragment indexes in parallel … meaning more than one at a time. I could not find one on the web so I built my own.

My solution will create a SQL Agent controller job and multiple worker jobs. The controller job will create a queue of indexes to be defragmented based on runtime parameters passed to it. Once the queue is created it will spawn the worker jobs which will proceed to take indexes off the queue until the queue is depleted or the termination window has expired.

The below table shows the performance gains we were able to get with our index defragmentation. Just using one job with Ola’s script took over 24 hours. When we broke it out into four jobs it still took 6-8 hours. With my script and six worker jobs the time went down to 3 hours for most all the indexes. There are two REALLY big indexes that take about 7 and 4 hours individually because the data in the table changes a lot daily. Finally, I realized that most of the indexes were set to a fill factor of 100%. Which means that I was constantly having high fragmentation on tables. So, I rebuilt all the indexes at 85% fill factor and the defragmentation time went down to about an hour per day with those two other indexes still being the outliers. I feel like that is a pretty good result.

ScriptJobs Duration
Ola Hallengren’s script1Over 24 hours
Ola Hallengren’s script46-8 hours
My script63 hours with 1 index at 7 hours and 1 at 4 hours
My script with fill factor at 85%6Monday: 1 hour (30 minutes of that is to build the queue) Other Days: 20 minutes (15 minutes of that is to build the queue)

The maintenance solution is release under the MIT License so feel free to use it as you wish just please let me know if you have found it to be helpful. You can click the button below to download the solution. The second link is to the source code repository.

https://bitbucket.org/davinstuder/index-maintenance/

‘Twas the Night Before Passover

  • by

When I was a student at Multnomah we had to do a final project for our Pentateuch class. I wrote this. I found it the other day as I was going through our filing cabinet and pulled it out to read to the kids for Passover. Enjoy!

‘Twas the night before Passover, when all throughout Israel’s foe
Not a creature was stirring, not even Pharaoh.
The Hebrews painted blood on their doorframes with care,
In hopes that the Spirit wouldn’t stop there.
The Egyptians were nestled all snug in their beds,
While visions of pyramids danced in their heads.
And Moses was sitting and talking with God,
About all the things he done with his rod.
Nine plagues he had brought on the land with a clatter,
But Pharaoh still didn’t know what was the matter.
Away he sent Moses each time with a flash,
Oblivious that again with God he would clash.
More rapid than eagles his curses they came,
And he whistled and shouted and called them by name.
Now, blood! Now, frogs! Now, gnats! And, flies!
Dead livestock! Boils! Hail! Some locusts! Dark Skies!
The tenth one reserved for the firstborn to fall,
God dashed away, dashed away, dashed away them all.
As dry leaves that before the wild hurricane fly,
All the nation of Egypt started to cry.
So, up from the houses the curses they flew,
At all of the Hebrews who’d known what to do.
In the twinkling of twilight they slaughtered a lamb,
Then cooked as commanded from the the Great I AM.
They ate with their cloaks tucked in at their waist,
And as God had commanded they did this with haste.
The bread that they ate couldn’t have any yeast,
If it did, it would certainly spoil the feast.
And bundles of gold and silver they pillaged,
Cause God made them favorable to all in the village.
Now Pharaoh was pondering “What should I do?”
“Cause if they don’t leave, I could die too.”
So he bid all the Hebrews and Moses to part,
But wouldn’t you know it God hardened his heart (again).
He sprang to his chariot, to his team gave a whistle,
Then his army flew like the down of a thistle.
They chased them until the Red Sea they arrived,
But Moses had parted the sea on both sides.
The Hebrews, they crossed the Red Sea without trouble,
So Pharaoh decided to try on the double.
He started to cross the Red Sea with his troops,
But then toward the middle looked up and said “Oops!”
The waters they started to tumble and fall,
Not one man survived, not one man at all.
Then Moses exclaimed, ere they walked toward the land,
“To God be the glory, we were saved by His hand.”

How to Download Your OverDrive History in CSV Format

  • by

The Background

As a family we read a ton. I started reading to my kids nightly when they were young and to this day we still have story time almost every night. It is a highlight of our family time together. As a result, my kids also love to read. This became an issue when we found ourselves visiting the library multiple times a week to get more books for my son. So, eventually we just started maxing out our library card. It was not unusual for us to have 50 books checked out. That, however, brings its own set of issues. When you have to return the books it can be a challenge to find them all!

Then we found out that our library partners with OverDrive and we can get books for our kids in e-book format with the click of a few buttons. (You should be hearing Handel’s Messiah playing in your head right now) Our world changed forever. Gone were the multiple trips to the library or searching high and low for missing books.

After much research we found out that we could get Amazon Fire tablets for our kids to use as e-readers (strictly e-readers) and we could load them up with e-books. Many of the books on OverDrive will let you check them out in Kindle format. Then, you simply use Amazon’s parent dashboard and you can share the selected Kindle books to your child’s tablet. It’s a bit of a wonky process, but it works pretty well. Now, we have a different issue entirely. Our kids are flying through books like crazy and I’m constantly researching for new series. Oh well, I guess it’s a good problem to have.

Downloading the history

Well, the other day I wanted to get the history of all the books we have checked out with OverDrive. To my dismay, there was not a way to do this. But, hey I’m a programmer! I can do hard things. So, I decided to work something up. At first I thought I was going to have to screen scrape the data from the OverDrive history page and programmatically next through every history page to compile it all. But then I noticed that when I clicked between the pages of history that the screen did not refresh, which suggested to me that a web service was being called. Voila! After looking through the networking calls being made by OverDrive I discovered their REST API and found out it is pretty easy to use and has a ton of information. So I set to work and here is the fruit of my labor. It is a browser bookmarklet. Simply drag the link below to your web browser’s bookmarks bar to create the bookmarklet. Then, when you are on your OverDrive history page click the bookmarklet. It will download all your history data and compile it into a CSV file that you can open in Excel or any other similar program. I hope it works well for you. Please leave a comment if you find it to be useful.

Bookmarklet:
Download OverDrive History <- drag this to your bookmarks toolbar

The code

Here is the code for the bookmarklet in case you are curious.

var OverDriveHistory = {
	baseURL: window.location.origin,
	restURL: '/rest/readingHistory?page={0}&perPage={1}&sortBy={2}',
	header: 'Title,Sub Title,Author,Series,Publisher,Publish Date,Star Rating,Star Rating Count,Matrity Level,ISBN,Cover Art URL,Borrow Date',
	csv: '',
	currentPage: 1,
	lastPage: -1,
	pageSize: 100,
	sortBy: '',
	error: false,
	init: function(){
		var t = this;
		if(t.isOverDriveHistoryPage()) {
			if(typeof jQuery == 'undefined') {
				t.getJavaScript('//code.jquery.com/jquery-latest.min.js', function(){
					t.start();
				});
			} else {
				t.start();
			}
		} else {
			alert('Please run this bookmarklet from your OverDrive history page.');
		}
	},
	start: function(){
		var t = this;
		var url = t.baseURL + t.restURL.replace(/\{0\}/g, '1').replace(/\{1\}/g, t.pageSize).replace(/\{2\}/g, t.sortBy);
		
		t.sortBy = jQuery('.AccountSortOptions-sort').val();
		t.showOverlay();
		
		jQuery.ajax({
			url: url
		})
		.done(function(data) {
			t.lastPage = data.links.last.page;
			t.csv += t.header + '\r\n';
			t.getData();
		})
		.fail(function() {
			t.error = true;
			t.finalize();
		});
	},
	getJavaScript: function(url, success){
		var script = document.createElement('script');
		var head = document.getElementsByTagName('head')[0];
		var done = false;
		
		script.src = url;
		script.onload = script.onreadystatechange = function(){
			if(!done && (!this.readyState || this.readyState == 'loaded' || this.readyState == 'complete')) {
				done = true;
				success();
				script.onload = script.onreadystatechange = null;
				head.removeChild(script);
			}
		};

		head.appendChild(script);
	},
	isOverDriveHistoryPage: function(){
		return window.location.href.toLowerCase().indexOf('overdrive.com/account/history') == -1 ? false : true;
	},
	getData: function(){
		var t = this;
		var url  = '';

		if (t.currentPage == t.lastPage + 1) {
			t.finalize();
		} else {
			url = t.baseURL + t.restURL.replace(/\{0\}/g, t.currentPage).replace(/\{1\}/g, t.pageSize).replace(/\{2\}/g, t.sortBy);
			jQuery.ajax({
				url: url
			})
			.done(function(data) {
				var isbn = '';
				for(var i = 0; i <data.items.length; i++){
					isbn = '';
					for (var x = 0; x < data.items[i].formats.length; x++) {
						if(typeof data.items[i].formats[x].isbn != 'undefined'){
							isbn = data.items[i].formats[x].isbn;
							break;
						}
					}
					
					//Title,Sub Title,Author,Series,Publisher,Publish Date,Star Rating,Star Rating Count,Matrity Level,ISBN,Cover Art URL,Borrow Date
					t.csv += t.escapeCSV(data.items[i].title) + ','
					+ t.escapeCSV(data.items[i].subtitle) + ','
					+ t.escapeCSV(data.items[i].firstCreatorName) + ','
					+ t.escapeCSV(data.items[i].series) + ','
					+ t.escapeCSV(data.items[i].publisher.name) + ','
					+ t.escapeCSV(data.items[i].publishDate) + ','
					+ t.escapeCSV(data.items[i].starRating) + ','
					+ t.escapeCSV(data.items[i].starRatingCount) + ','
					+ t.escapeCSV(data.items[i].ratings.maturityLevel.name) + ','
					+ t.escapeCSV(isbn) + ','
					+ t.escapeCSV(data.items[i].covers.cover510Wide.href) + ','
					+ t.escapeCSV(data.items[i].historyAddDate)
					+ '\r\n';
				}

				t.currentPage += 1;
				t.getData();
			})
			.fail(function() {
				t.error = true;
				t.finalize();
			});
		}
	},
	escapeCSV: function(value){
		var t = this;
		var newValue = value;
		
		if(!newValue){
			newValue = "";
		} else {
			newValue = newValue.toString();
		}
		
		if(newValue.indexOf('"') != -1 || newValue.indexOf(',') != -1 || newValue.indexOf('\r') != -1 || newValue.indexOf('\n') != -1){
			newValue = '"' + newValue.replace(/"/g,'""') + '"';
		}
		
		return newValue;
	},
	showOverlay: function(){
		var html = '';
		
		html = '<div id="history-fetch-overlay" style="position: fixed;width: 100%;height: 100%;top: 0;left: 0;right: 0;bottom: 0;background-color: rgba(0,0,0,0.5);z-index: 1000;"><div style="position: absolute;top: 50%;left: 50%;font-size: 50px;color: white;transform: translate(-50%,-50%);">Fetching data. Please wait.</div></div>';
		
		jQuery('body').append(html);
	},
	removeOverlay: function(){
		jQuery('#history-fetch-overlay').remove();
	},
	finalize: function(){
		var t = this;
		if(!t.error) {
			var fileName = "OverDriveHistory.csv";

			if (window.navigator.msSaveOrOpenBlob){
				// IE 10+
				var blob = new Blob([decodeURIComponent(encodeURI(t.csv))], {
					type: 'text/plain;charset=utf-8'
				});
				window.navigator.msSaveBlob(blob, fileName);
			} else {
				var pom = document.createElement('a');
				pom.setAttribute('href', 'data:text/plain;charset=utf-8,' + encodeURIComponent(t.csv));
				pom.setAttribute('download', fileName);
				document.body.appendChild(pom);
				pom.click();
				document.body.removeChild(pom);
			}
		} else {
			alert('Something went wrong. Please try again.');
		}
		
		t.removeOverlay();
	}
};

OverDriveHistory.init();

Amazing Vanishing Lego Wall

  • by

My son, who loves Lego, built a pretty awesome creation the other day. It is a super cool cave scene with a hidden surprise … a vanishing wall. He designed and built it all himself and I was so impressed I just had to upload a video of it.

FUME – A Better Confluence User Macro Editor

  • by

One of the best and most overlooked aspects of Confluence, both by Atlassian and Confluence administrators, is user macros. There are so many useful scenarios for user macros. Here are some:

  • Templated snippets
  • Overriding built in macros example with task list report
  • Quickly creating your own macros
  • Inserting arbitrary html/css/javascript into a page without having to enabled the html macro

However, there are some big usability issues with the user macro editor. First it’s super easy to accidentally delete one. The delete link is right next to the edit link and seriously, there is no confirmation on the delete link. It’s just gone. Ack!

Second, the link to create new user macros is at the bottom of the page. If you have more that what can fit on a screen you have to scroll down to get to the link to create a new one … this just gets worse over time as you create more.

Third, the template box in the editor is just a plain old text area … no line numbers, not syntax highlighting, it’s not even a mono-spaced font! Grr!

Fourth, the cancel button doesn’t ask you to confirm canceling the edit if you have made changes to the user macro and since it sits right next to the save button it’s easy to miss. Hope you can recreate your work quickly.

Finally, every time you save it kicks you back to the list page. So, if you want to make some changes and try it out on a page you have to click back into the editor every time you save and whoops you accidentally just clicked delete instead of edit! There goes all that work.

So, without further ado … FUME. Fantastic user macro editor. The fantastic part is really just because I needed a word that ended in “ume” and that was the only word I could think of. Really it’s not all that fantastic … maybe just great, but gume isn’t even a word. Then I thought “How about  great looking user macro editor”, but that would be glume and … well … yeah, that kinda defeats the purpose. So, FUME it is. All in all I think it is a much better editing experience than the default setup. Here are some of the features:

  • Copy that “Create a User Macro” link to the top of the list page … no more scrolling
  • Delete confirmation on the list page
  • Template box changed to a source code editor with (Ace editor):
    • monospaced font
    • line numbers
    • syntax highlighting
    • find and replace
    • code folding
    • column select
  • Confirmation on cancelling edits of the user macro if the template has been changed
  • Asynchronous user macro saves
  • It will do your dishes and laundry … ok, not quite yet

Update 4/9/2018:

Ignore the “How to Setup” section below. I’ll leave it there, however, for the sake of continuity. I decided to package this up as an add-on in the Atlassian Marketplace. I named it Enhanced User Macro Editor (EUME … pronounced you-me … it’s a stretch I know). It seemed a bit more humble of a name and is more descriptive of what it is. I hope it is as useful for you as it has been for me. Marketplace link below.

Enhanced User Macro Editor

How to Setup

  • Download these CSS and Javascript files. (right click the links and choose “Save link as”)
  • Place them on a web server where they will be web accessible to your user macro editors.
  • Add this to the end of Confluence Admin -> Custom HTML -> At end of the BODY
<!--
*****************************************
* Fantastic User Macro Editor           * *****************************************
-->
<link rel="stylesheet" type="text/css" href="http(s)://{your server}/path/to/fume.css">
<script src="http(s)://{your server}/path/to/fume.js" type="text/javascript"></script>
  • Enjoy editing your user macros.  🙂

Screenshots

User Macro List

User Macro List Page

User Macro Template Editor

User Macro Template Editor
Splunk HL7

Parsing HL7 with Splunk

  • by

At my job I do a fair amount of work with HL7. If you work in the medical field you probably know that HL7 is the language that medical systems use to talk with each other. It’s a fairly simple format that uses carriage returns and pipes to delimit fields … ok there are a few other delimiters as well, but the carriage returns and pipes are the big ones. Below is an example HL7 message.

MSH|^~\&|MegaReg|XYZHospC|SuperOE|XYZImgCtr|20060529090131-0500||ADT^A08|01052901|P|2.3
EVN||200605290901||||200605290900
PID|||56782445^^^UAReg^PI||KLEINSAMPLE^BARRY^Q^JR||19620910|M||2028-9^^HL70005^RA99113^^XYZ|260 GOODWIN CREST DRIVE^^BIRMINGHAM^AL^35209^^M~NICKELL’S PICKLES^10000 W 100TH AVE^BIRMINGHAM^AL^35200^^O|||||||0105I30001^^^99DEF^AN
PV1||I|W^389^1^UABH^^^^3||||12345^MORGAN^REX^J^^^MD^0010^UAMC^L||67890^GRAINGER^LUCY^X^^^MD^0010^UAMC^L|MED|||||A0||13579^POTTER^SHERMAN^T^^^MD^0010^UAMC^L|||||||||||||||||||||||||||200605290900
OBX|1|NM|^Body Height||1.80|m^Meter^ISO+|||||F
OBX|2|NM|^Body Weight||79|kg^Kilogram^ISO+|||||F
AL1|1||^ASPIRIN
DG1|1||786.50^CHEST PAIN, UNSPECIFIED^I9|||A

Each line in the message is called a segment and each segment can be divided into fields based on the pipes. For instance the third line is the PID segment which has patient information such as the MRN (PID 3), patient name (PID 5), birth date (PID 7), etc. The PV1 segment has information that relates to the patient visit. It is a fairly concise format without much overhead and as such is perfect for medical institutions where these kinds of messages are flowing constantly throughout the day.

The Problem

In a typical medical environment there will be a system called the HL7 routing engine that serves as an intermediary between all the various medical systems in the clinic or hospital. The HL7 engine can route messages to one or various systems and transforms them en-route based on rules. Most HL7 engines have the ability to log the messages sent through them in some format.

Often times there may be a need to lookup what messages were sent to various system to troubleshoot problems. In many cases there is no great means of searching through the thousands or even hundreds of thousands of messages sent each day to troubleshoot these issues.

The Solution

About a year ago I was approached by some folks from Splunk about creating a Technical Add-on (TA) for Splunk for parsing HL7. After many months of working with one of their engineers named Joe Welsh we were able to release the free ‘HL7 Add-On for Splunk“. We tested the add-on by throwing millions of our HL7 messages at it to make sure it parsed the messages correctly.

With this TA we can have Splunk monitor our HL7 logs and in real-time are able to quickly search those logs to troubleshoot issues, report on failed messages, and view dashboards to monitor the health of our HL7 environment. I have been super pleased with the results.

If you are a medical institution that uses Splunk check out the add-on … it’s free. See what awesome things you can do with Splunk and HL7. Let me know in the comments if you have found it useful.

Splunkbase Link
https://splunkbase.splunk.com/app/3283/

Powered by WordPress, running on raspberries

  • by

The Issue

My Raspbery Pi running WordPressMeet my new web server! Isn’t it cute? Up till recently I have been running my website on my home machine. This has worked fine as this site doesn’t get a ton of traffic and my home machine is fairly beefy. However, I’ve always been slightly leery about running my web server on my main computer just for security reasons. If it were to get hacked I could be potentially opening up everything to the hackers. Not only that, in order for the site to be up I had to leave my home machine running 24/7. This is not great on the electricity bill. Also, during the summer months in our not so air-conditioned house it actually adds significantly to the heat. Which is not too fun for my wife who has to been home all day in the heat while I sit at work in the air-conditioning.

The Solution

So, I started looking around for a not too expensive solution to this. As I researched I started seeing more and more people running WordPress on a Raspberry Pi. Now, I can muck my way around Linux, but it’s certainly not my fortĂ©. Fortunately, I found a great tutorial for installing WordPress on a Raspberry Pi that even a Linux noob like me can follow.

It worked great! I was able to get WordPress set up, move my content over and flip the port forwarding over to the Pi within a few hours. I also set up a dynamic DNS client on the PI to keep my address current with my domain registrar. I now have a separate server, that runs on pennies, puts out little to no heat, is completely quiet, and cost very little to purchase. I could not be happier.

Bookmarklet for Creating a CSV From an HTML Table

We have a wiki at work that we use for documentation called Confluence. It has the ability to export pages to Word or PDF, but a lacking feature is the ability to take a table and export it for Excel. So, I decided to create a bookmarklet that will allow you to select an HTML table on any web page and create a CSV file from that table that can be downloaded to your machine. Below is the bookmarklet. Just drag it to your bookmarks toolbar. Then give it a shot by clicking the bookmarklet and the below example table should get a link right before it that says “Export to CSV”. Click that link and you will be prompted to download the CSV version of that table. Let me know if it is useful for you.

On thing of note, this will not work correctly in Internet Explorer 9 and below as IE will not allow data uri’s for anything other than images.

Bookmarklet:
Export to CSV <- drag this to your bookmarks toolbar

Update:
05/08/2019 – Enhanced to not kill line breaks.
09/16/2016 – Now works with IE 10+.
04/19/2016 – Fixed issue with tables that have header cells not on top.

Example Table:

Heading 1Heading 2Heading 3Heading 4Heading 5Heading 6
Data 1, 1Data 1, 2Data 1, 3Data 1, 4Data 1, 5Data 1, 6
Data 2, 1Data 2, 2Data 2, 3Data 2, 4Data 2, 5Data 2, 6
Data 3, 1Data 3, 2Data 3, 3Data 3, 4Data 3, 5Data 3, 6
Data 4, 1Data 4, 2Data 4, 3Data 4, 4Data 4, 5Data 4, 6
Data 5, 1Data 5, 2Data 5, 3Data 5, 4Data 5, 5Data 5, 6
Data 6, 1Data 6, 2Data 6, 3Data 6, 4Data 6, 5Data 6, 6

Bookmarklet Source: 

javascript:(function(){
    function getJavaScript(url, success) {
        var script = document.createElement('script');
            script.src = url;
        var head = document.getElementsByTagName('head')[0],
            done = false;
        script.onload = script.onreadystatechange = function(){
            if (!done && (!this.readyState || this.readyState == 'loaded' || this.readyState == 'complete')) {
                done = true;
                success();
                script.onload = script.onreadystatechange = null;
                head.removeChild(script);
            }
        };
        head.appendChild(script);
    }
    function addCSVLinks() {
        jQuery('.csvLink').remove();
        
        jQuery('table').each(function(index){
            jQuery(this).attr('data-csvtable', index).before('<a href="#" class="csvLink" data-forcsvtable="' + index + '">Export to CSV</a>');
        });
        
        jQuery('.csvLink').click(function(){
            var text = '';
            var csvTableIndex = jQuery(this).attr('data-forcsvtable');
            jQuery('table[data-csvtable="' + csvTableIndex + '"] tr').each(function(){
                jQuery('td, th', this).each(function(index){
                    if(index != 0) {
                        text += ',';
                    }
                    text += '"' + formatedText(jQuery(this).html()) + '"';
                });
                text += '\r\n';
            });
            jQuery('.csvLink').remove();
            downloadCSVFile('TableExport.csv', 'text/csv', text);
        });
    }
	function formatedText(html) {
		var ret = html;
		
		//replace line breaks
		ret = ret.replace(/\n/g, ' ');
		
		//replace tabs
		ret = ret.replace(/\t/g, ' ');
		
		//replace multiple spaces
		ret = ret.replace(/\s+/g, ' ');
		
		//Fix html encoded characters
		ret = decodeHtml(ret);
		
		//Deal with lines breaks and paragraphs
		ret = ret.replace(/<br>/ig, '\n<br>');
		ret = ret.replace(/<br/ig, '\n<br ');
		ret = ret.replace(/<p/ig, '\n<p ');
		
		//Deal with quotes
		ret = ret.replace(/"/ig, '""');
		
		//Deal first character being line break
		ret = ret.replace(/^\n/, '');
		
		//Remove HTML tags
		ret = ret.replace(/(<([^>]+)>)/ig,"");

		return ret;
	}
	function decodeHtml(html) {
		var txt = document.createElement('textarea');
		txt.innerHTML = html;
		return txt.value;
	}
    function downloadCSVFile(filename, mime, text) {
        if (window.navigator.msSaveOrOpenBlob){
            // IE 10+
            var blob = new Blob([decodeURIComponent(encodeURI(text))], {
                type: 'text/csv;charset=utf-8'
            });
            window.navigator.msSaveBlob(blob, filename);
        } else {
            var pom = document.createElement('a');
            pom.setAttribute('href', 'data:' + mime + ';charset=utf-8,' + encodeURIComponent(text));
            pom.setAttribute('download', filename);
            document.body.appendChild(pom);
            pom.click();
            document.body.removeChild(pom);
        }
    }
    if(typeof jQuery == 'undefined') {
        getJavaScript(
            '//code.jquery.com/jquery-latest.min.js',
            function(){
                addCSVLinks();
            }
        )
    } else {
        addCSVLinks();
    }
})();

Microsoft, the 90’s wants its security policy back!

  • by

Today, I went to change the password on my Microsoft account from a shortish hard to remember and type password to a nice and secure long and easy to type passphrase. With all of the hubbub these days around hacked accounts I want to make sure that my accounts are nice and secure. I just wish Microsoft was on-board with that.

Phrase1I went to my account and put in a pass phrase that seems pretty decent to me. Here is the first message I got. It seems that my nineteen character passphrase is too long. Why is this an issue I ask? Why does Microsoft care if I want to have a long password? If I want to take on the burden of typing those extra characters what is it to them? This is not a limitation with any other Microsoft system I have ever used (personal machine and work machines). I’m sure their own systems are based on their own stack, so this is an artificial limit that they have put in that has no real value except to make my account easier to hack.

Phrase2Ok, since I can only have up to sixteen characters in my password I guess I’ll have to comply. Sixteen characters is still pretty long right? So, I swapped out some words and made a few tweaks only to be presented with this. REALLY!? By the way, the character that they do not like is a space. So, I thought I would see what characters they do allow. They allow A-Z, a-z, 0-9, and every special character printed on your keyboard … just not space. Ugh!! It seems they are going out of their way to discourage pass phrases when it has been shown time, and again, and again, and again that pass phrases are more secure than passwords.

So, in the end, I am forced to bow to their ridiculous security policy that excludes one character and forces me to 16 characters. What do you think?

Confluence Profile Photo Uploader

  • by

Small disclaimer: this works great on my setup. I have not tested it outside of my setup. It should work just fine as I am using the standard API’s that come with Confluence, but it is an open source project and I’m not getting paid for it, so I haven’t done extensive testing in all scenarios.

At the Vancouver Clinic we use a wiki product called Confluence. It is a fantastic product that is super customizable via add-ons, extensions, custom html/js/css, and SOAP and REST APIs.

Unfortunately, Confluence does not have a way for administrators to bulk upload user profile photos into it. For corporate organizations that want to make sure that user photos are uniform and professional this is somewhat cumbersome. As a result Confluence may have user profiles with Mickey Mouse, Wolverine, or worse as the photo … don’t get me wrong Wolverine is pretty cool, just not professional.

Because of this I decided to write a utility that allows an administrator to sweep in photos in a specified folder with a file mask of “%username%.%extension%” into Confluence. The utility will accept .jpg, .jpeg, .tif, .tiff, .png, and .bmp files. Optionally, you can specify a folder to archive the swept-in photos to after they have been uploaded. Confluence profile photos are 48×48 pixels and since most photos are not square the utility will attempt to do face detection and crop the photo to the largest face in the photo. If the photo is already square you can opt to have the utility not perform the face detection. I am personally not smart enough to write face detection algorithms, so I am using EMGU/OpenCV to do the face detection (which is why the compiled version is so freakin big). Hey, I got no problem standing on the shoulders of giants.

For automation purposes there is a command line switch /headless that will do the upload without any GUI using the configured settings. This works great with Windows Task Scheduler.

This is working awesomely for the clinic and fortunately my manager has allowed me to open source the code. So, I have uploaded it to Bitbucket for anyone to use, or make fun of my code. : ) Seriously, have at it if it will help your organization. And by all means let me know if there you run into any bugs along the way.

https://bitbucket.org/fredclown/confluence-profile-photo-uploader/