This End Up

WARNING: This is probably going to be my yearly rant blog posting for 2012.  Feel free to just walk away.

Ok, I'm pretty mad right now.  I don't want to write this blog post, but I just can't believe what's going on and what I'm being told to do.  This should not be happening and I want to do all that I can to make sure this sort of thing does not happen again. I'm reminded of the Holy Discontent talk at 2005 Leadership Summit that started me on the path I'm now on.

Let's back up a few months.  We were told we had to upgrade from our mostly stable Arena 2010.1.x to 2011.1.500 because the background check service provider that we use with Arena was changing on 3/1. Up against the wall, we reluctantly upgraded.  A month later (3/29) I got an email from Arena saying a known bug (in their new code; the code that was supposed to be the fix for the "problem") would be fixed in an upcoming 2011.2.400.  Herein starts the real topic of this post.  Why was I being told a bug fix would be put into a new release of the product and not a .600 patch fix?  Semver (Semantic Versioning) is all but the standard way of doing things now, and even if you don't version using the X.Y.Z form, every software vendor knows you issue patches to fix bugs.  You don't ask your customers to upgrade to a newer version (which in almost certainty) has it's own set of bugs and changes to deal with. Me and others complained, and today I was told the vendor is now going to create a .600 patch -- it should be ready in another month or so (around the end of May?).

What?  Two+ months to issue a patch? Are you kidding me?  What is this patchy philosophy?

A patch from a paid vendor need not take this long to issue.  I understand QA testing for a full release takes time, but a full QA/test cycle is not needed for a patch because you should be changing only one piece of code and testing that one piece of code.

In the meantime, several times a week (we've had 10 failures out of approximately 30 requests in the past month) I was asked by the vendor to have our ministry assistant using the following steps to work around the problem:

Steps for identifying and retrieving outstanding background checks containing Alerts

1)      Run the agent configuration utility on the Arena webserver and click on the background check agent. Go to the notifications tab and identify the email address that alerts are set to go to.

2)      Access the noted account and look for an alert email containing the information shown below.
<IMAGE REMOVED>

3)      If such an email is found, they will need to contact Melissa ****** at ABC @ 800-***-**** ext. *** or ***@serviceprovidervendor.com. Request that she email them the Name and URL of all individuals with an Alert on their record. They should receive a url for each person that looks something like: https://blahblahblah/foo/bar?ReportID=**********.

 4)      Once they receive the info from ABC, they would then do the following (access to Arena and SQL required):

1.       In Arena, go to the affected background check and click on it. In the url it will provide you with the check id, take note as that’s the id in the arena database. Repeat for each check in the list that ABC provided.

2.       Go to SQL and run:

select background_check_id,status,result,results_link from you_have_got_to_be_kidding_me where background_check_id = x (where x is the id number of the check). Only the id and status should have a value, result and result link should be blank.

3.       Run:

 update foo_bar_blah_check set status = 4, result = 'Complete - Record',results_link = 'URL that ABC provided'

where background_check_id = x

5)      After they have completed step 4, they can either manually stop and restart their agents to pull the background checks now, or they can wait for the agent to cycle and pull down the background checks.

6)      To determine if the background check record has been successfully retrieved, run:

select background_check_id,status,result,results_link, results_blob_id from more_fun_with_sql_tables where background_check_id = x

The check has been successfully retrieved if there is a value in the results_blob_id field, note the value and proceed to step 7.

7)      To confirm the existence of the background check document, run:

select created_by,modified_by,file_ext,mime_type,original_file_name from big_blobby_table where blob_id = x  (x being the value noted in step 6)

And to complete the process, updating the document so that it can be viewed, run: 

Update that_big_blobby_table_again  Set file_ext = 'pdf', Original_file_name = 'BackgroundCheckReport.pdf' Where file_ext like 'cfm?reportid=%'.

 * I've changed certain tables/URLs/names for obvious security reasons and to protect the mostly innocent.

If I had the source code, I would probably just fix this myself instead of writing this stupid blog post. I realize I'm setting the Rock ChMS initiative which I've joined to meet these same expectations -- even though it's is a free and open source product.  I just don't think it's reasonable to have to run broken in a broken state for two months while a patch is being assembled.

An Idea

A few months ago Kim Vehon, from our Worship team came, into our office to share a video showing a bunch of phones hanging from wires flashing on and off. It's pretty cool. She wondered if we could do something like that during our Christmas services at Central. Building a "fat-app" to control a bunch of phones wired together did not seem like a challenge nor interesting enough to be worth pursuing.

A Better Idea

But then we started thinking. What if we tried to control the congregation's phones and what if we used only their existing 3G network connection?  And, since hardly anyone would want to install an app from the market/store, what if it only used their phone's web browser?  We also thought it should be built in such a way as to reduce the dependency on the network -- in other words the phone should get what it needs from the server and then be able to loose network connection without impact to the performance. Game on. I agree with Jason (tweet), this might be one of the funnest things I've worked on in a long time. Jason and I call our software CrowdSync...

Ultimately we also needed to generate a lights time coded "track" from a Christmas song (midi file) with each note being assigned to 1-4 colors.  A person's phone would receive the track, be randomly assigned one of the four colors, and start playing the track in sync with the band.  Sounded simple enough. Thankfully I found the C# MIDI Toolkit code from Leslie Sanford that reads midi files (thank you Leslie!) which I was able to modify in order to extract and generate our time coded light track as JSON data which looks roughly like this:

{ 
    startTime: 123578916,
    endTime: 12345667,
    ticks: [
                { notes: ["a"], time: 27, duration: 900 },
                { notes: ["b"], time: 982, duration: 900 },
                { notes: ["c"], time: 1940, duration: 900 },
                { notes: ["d"], time: 2908, duration: 900 },
                ...
           ]
}

The Time Problem

Pretty quickly we eliminated web sockets and other client-server signaling technology for a variety of reasons including lack of consistent device support, chattiness, lag, and timing control.  We wanted to constrain ourselves in order to force us to think differently about certain problems -- such as the "how do we get all of these devices/clients to start at exactly the same time?" problem.

Initially we were thinking we could rely on the time from the phone's operating system.  You might think two 3G Verizon phones would have the same time, right?  Wrong. Very wrong. We saw phones that were off anywhere from several seconds to a few minutes. (Who knows how or where each phone is getting its time from? It doesn't appear they're using a common Network Time Protocol (NTP) server.)

After some medium scale client tests and experimentation, we came up with the following approach:  Each client asks our server for the current time, calculates the delta (from it's local time), and repeats this about 20 times over the course of about 20 seconds.  The error introduced because of network latency is reduced to a minimum because we use only the 'smallest' delta from our samples.

Once we've got the correct delta we calculate and shift the entire track's note times to represent the exact localized time each particular note should play (light up) on that device.  Then we basically wait until the note's time occurs and set some CSS to play that particular note's light.

Other Stuff

Since not all browsers can handle HTML5, we had to keep things simple and used basic HTML, CSS and JavaScript (CoffeeScript).  Other problems worth mentioning are: cell phone auto dimming, screen locking, and mysterious time lost as a result of certain mobile phone browser interruptions - such as screen lock and wake up.  (On my Android phone, a screen lock and re-open would cause a simple JavaScript clock to become out of sync with correct time.  It smells like a timing bug with the OS - but what do I know.)

Forward to November when it was decided that our system should also power the worship center's IMAG side screens with something cool, show an event count down timer, and the drive the band's click track.  That ended up being a real blessing because we got to work on some really fun stuff.  After more experimentation for the "something cool" part, and more ramping-it-up ™ with statements like "what if we could ...", we ended up programmatically building a Christmas Tree as colored circles on the HTML5 canvas to match our church's Luminous graphic.

We somewhat randomly plot circles in the shape of a Christmas tree using a little Math.sin() trick to get the curve we wanted.  (Remember that stuff you learned in high school -- it really does come in handy!)  The same JSON encoded light track data would be used to control the four sets of lights on the tree.  We kept the same code base and ended up with some configuration settings to control whether the simple HTML/CSS (cell phones) or HTML5 canvas w/tree was being used (side screens).

Ultimately, the midi file click track turned into a full blown amazing score and arrangement of Carol of the Bells by Adrian Darsee in mp3 format.  There was more fun getting the audio track to play which perhaps Jason will cover in his blog.

We also created a small admin panel as an Arena module so that the A/V tech guys at each campus could store the event start time for that service once they knew exactly when they wanted it to start.

Time Shifting Gotchas

Late in the game we discovered something unfortunate when we put it all together with all the other equipment (QLab, the worship center's A/V systems and side screens, a full 30 minute pre-service countdown, etc.).  Not all seconds are created equal. I should have realized this, but somehow it failed to register.  Due to NTP being used on the various systems, the QLab mac, the mac running Chrome for the IMAG, and our server would slowly drift by a 1 to .5 seconds during the 40 minute countdown.  That's a big problem if you're trying to achieve millisecond synchronization.  We turned off NTP as a work-around but realize it's something we want to address in future versions of CrowdSync.  We're also going to implement all the server side stuff in Node.js.

Conclusion

The goal was for each person to "be the light" and "let your light shine" as a literal and symbolic expression during the worship to our King. Overall it was a success for the whole opening worship experience but when the congregation erupted into loud applause at the end I knew they enjoyed the experience too.  I hope we have some of it captured on video.

We'll post the latest alpha CrowdSync source up on Github shortly.

Today marks the launch of our 5th Arena ChMS powered website.  This time around it was more of a cosmetic face-lift, and as far as the switch-over was concerned, we used all the tricks we learned last time to make the change very smooth (more on that below).  The bulk of the switch took 8 seconds (scripted), about 1 hour of minor manual tweaks by Jason Offutt and I, followed by roughly 4 hours of additional fine tuning by Jason Ake.

A few lessons were learned during the 4th generation site (aka Hasselhoff) which were largely based on feedback we received from the congregation and staff.  Gone is the campus selection integration that was initially prevalent after the initial launch in the summer 2010.  Now the only place a person needs to think about which campus they care about is when they're exploring the calendar, giving online (since one of our campuses has a different giving system - ugh), and when they simply want information about our different campuses.  It mostly turns out that people don't like campus separation/segregation.

Although the new site is based on the "Elegance" theme, props go to Jason Ake (our new web designer) and our graphic artists, Jeremy Wagner and Mitch Eiler, for their many cool elements and tweaks.  Most of the cool code changes including the newly tweaked event calendar, our Facebook login integration, and Vimeo video wall come from Jason Offutt, who's continually pushing me to learn the latest techniques and libraries.  To that end, this time around we used a little Mustache.js, Backbone.js, and as we discussed at RefreshCache 2011, we wrote some of our JavaScript in CoffeeScript.  Our promotion slider (seen here) is still based on our Promotions via XSLT module but this time around our XSLT spits-out Awkward-Showcase jQuery library goodness to handle the transitions and widgety-thumbnaily UI.  The recent blog entries that are pulled into the the home page is using a slightly customized (I added caching) version of Arena's standard XML File Transformation module (which I just posted to the shared repo here).  Lastly, we also used the Promotion via XSLT module on a standalone page to "feed" our Arena based promotions into our "Welcome" Facebook page as seen here:

Overall these changes took our code team between 2-4 weeks of execution time while all the rest of the project work took Jason Ake between 2-3 months of planning and execution.

Now, about the planning, staging and cut-over...

First of all we decided to ease our pain by creating four new Arena templates which would roughly match our previous four (home page, child page, single column page, and wide page). In addition to that, whenever possible, we used identical "area" (placeholder) names inside the templates.  These two steps alone simplify the cut-over tremendously since our cutover SQL script now basically only has to change (for example) pages that use template 65 and update them to use template 77, etc.  I highly recommend you use this same technique when upgrading/updating your website from generation to generation.

Once the templates were created/beta we also immediately added them to our production Arena install so that we could get their final templateID (for use later when writing the cut-over script).  At that point in the process it's also a good idea to add any new pages that you're going to want in the new website and just set the "Display in Nav" to false.  On the day of the cut-over your script can surgically flip that bit to make it show up.  At about t-minus 2 weeks we copied our production Arena database and Arena website folders into a new test instance for testing the cut-over scripts. We added an alias to our internal DNS and set up IIS to serve the test site up as http://testweb/ for general staff User Acceptance Testing.

Now that this website is behind us, next up will be creating a personal, personalized space on the site and the modules needed for people to access all of "their" stuff (profile, prayer requests, small groups, contribution statements, event registrations, customized news/promotions, etc.).

I've joined forces with an initially small team of developers and artists to form the Spark Development Network (www.sparkdevelopmentnetwork.org) and we are creating a new, open source Church Management System (ChMS) called Rock ChMS. You can read the press release if you wish.

This may come as no surprise to some of you after hearing me last year at RefreshCache (www.refreshcache.com) and previous rants on why open source is the best option.  Last week I reviewed my "State of the Union" presentation from last October.  Reading that now, you might think that Rock ChMS was already underway; however our group had not even had a single discussion about anything remotely related.  Over the years several groups tried to apply pressure to our vendors to release open source versions of their product and I think we waited as long as we could, but in the spirit of RefreshCache, decided to just make it happen on our own.
 
Although everyone at Spark Development Network will have slightly different reasons, basically we wanted to collaborate on a framework that was free for the Christian Church and para-church community, that's beautiful, easy and simple to use, easy to administrate, open source and easy to develop against for the church developer community.  For those who want to eventually run their church on Rock ChMS, should they ever encounter a mission critical bug, their developer can jump in and fix it without having to wait for an official patch.

Rock ChMS is only a pre-alpha seed at the moment, but the source is now open and publicly available on Github: https://github.com/SparkDevNetwork/Rock-ChMS. This was done so that others could collaborate and get involved from the earliest days of the project.  Rock ChMS is going to be a full featured Church Management System built on top of a custom CMS application framework, so you may find it's similar to other CMS frameworks you've used in the past. It's an ASP.NET 4.0 Entity Framework application written in C#, so if you’re serious about wanting to help The Church, then git on over to GitHub and fork the repo. When you're ready, submit a pull request and we'll take a look at your work. If you're interested in knowing more or want to get involved in other ways, sign up on our Stay in Touch page.

Something I've been meaning to have added to the Arena Custom Module Developer (ACMD) guide is this little known feature that was introduced at some point in the past few years.  It's a way to group your custom module's settings as seen here:

The effect is subtle, but when you have 20 or so settings like we have in our check-in module it starts to become really necessary (of course we haven't yet added groupings, but we will in the next release).

To create these groupings (aka categories) all you need to do is include the System.ComponentModel in your using section:

using System.ComponentModel;

 ... and then define a Category attribute with a grouping name argument for each module setting as seen in this example:

// Styling Group
[TextSetting( "Search Button Image Path", "Relative path ...", false ), Category( "Styling" )]
public string SearchImagePathSetting { get { return Setting( "SearchImagePath", "", false ); } }

[TextSetting( "Search Button CSS Class", "CSS classname...", false ), Category( "Styling" )]
public string SearchButtonCSSClassSetting { get { return Setting( "SearchButtonCSSClass", "", false ); } }

[TextSetting( "TextBox CSS Class", "CSS classname ...", false ), Category( "Styling" )]
public string TextBoxCSSClassSetting { get { return Setting( "TextBoxCSSClass", "", false ); } }

// No group defined -- these go into a "General Settings" section automatically
[NumericSetting( "Return Results Page Size", "The number of ...", false )]
public string ReturnResultsPageSizeSetting { get { return Setting( "ReturnResultsPageSize", "10", false ); } }

This year we had a little more than 1750+ kids use the automated check-in system for VBS across three of our campuses.

Looking at this graph which shows the number of check-ins per minute, you can see check-in does not ramp up as sharply as we would want on Monday (red).

(click for larger image)

By Tuesday (blue) after some adjustments were made and parents knew the routine a bit better, check-in ramps up quicker to about 50-60 kids per minute and 28 minutes later the lines were gone and the majority of kids were checked in.

It's OK, but not great. Ideally, you'd like to check in 1500+ kids per minute and have check in last one minute.  Not realistic or necessary since not all parents arrive exactly on time.  Perhaps 150 kids per minute is a goal.  Then check-in only lasts 10 minutes.  What will it take to reach that goal?

With some analysis, we found that it takes about 15-18 seconds per parent to check in their kids.  It turns out, much of the time it takes for someone to check in using our system involves punching in the phone number (see this video).  That's something I already knew, and it's one of the reasons why I wanted to keep barcode scanners at all our campuses. (a battle I lost -- for now).  If a parent uses a barcode they shave 6-12 seconds off that time (depending on how slowly a person normally types in their phone number) and they can check in their kids in about 4-6 seconds.

So if we don't start using barcode scanners again, I think the only thing we'll be able to to is drop in more, cheap, iPad kiosks... unless we start letting people check in using their mobile phone when they arrive on campus...

Our current church management system has been falling behind.  The Visual Studio precompiled website solution we get with the SDK still targets the .NET Framework 3.5 and this morning I ran into my biggest issue to date:

I can't use the NuGet.Core library in my new module until our current vendor either updates the solution to target the .40 framework or I figure out a way to get around this hurdle.  I don't think I can safely just change the build target on the solution since it's precompiled...  hmm.

Once I figure that out, I'll move onto the other problem.  The system is still using jQuery 1.3 (circ. 2009).  Yes, jQuery 1.4 was released at the beginning of 2010, 1.5 at the beginning of 2011, and 1.6 was just released last month.  I'm beginning to understand why President Roosevelt uttered the words "the only thing we have to fear is fear itself."

Any suggestions?

When you use a web based kiosk for your attendant-less check-in system, you should think about what's going to happen if your network fails you -- especially if you're using the 3G network.  This is what you'll see if the kiosk cannot reach the server:

That page is especially problematic if you don't have a keyboard because there is no way to refresh/reload the page if the network starts working again.

Luckily you can customize this content... it's just a pain to do it but I'll show you how.  Here's what our page now looks like:

The button is a nice feature because it's basically just going to load our check-in system page again.  Without that, in the past someone would have to reboot the whole system (pressing F5 on an stashed-away emergency keyboard would have also worked).  It's also worth noting that since you can use javascript in there, you should even be able to do something more sophisticated such as automatically retry every few seconds, etc.

Here's how we did it:

1. Download a Resource editor such as Resource Hacker. I read that it is possible to edit .mui files using VisualStudio, but after about a half hour of fail, I found Resource Hacker and it did it quite easily.
   
   
2. Grab a copy of your kiosk's ieframe.dll.mui file (found under C:\Windows\system32\en-US\) and stow it away for backup purposes.
   
   
3. Use Resource Hacker and edit the ieframe.dll.mui file.  Navigate down into the "23" folder (aka HTML folder) and select the DNSERROR.HTM item like so:
   
   
   
   
4. Make your changes.  It's just HTML, so you know what to do...
   
   
5. Press the "Compile Script" button.
   
   
6. Save the file and quit.
   
   
7. Copy the file over to your kiosk's C:\Windows\system32\en-US\ folder.  If you're unable to, because the file is in use or similar, just rename the live file (oddly Windows let's you do that) and then copy the file down to that folder.
   
   
8. Start IE and kill your network to see the result.
   

[note: This is an especially 'angry coder' post -- so I encourage the the faint of heart reader to simply skip this article.] 

Although I hate to admit it, I know there is no such thing as bug free software releases.  With testing and a solid QA process in place, you most definitely can drive software defects in new releases toward zero. However, the more complex the software is, the less likely it will come close to zero.  Some have argued from a business perspective it's not cost effective to even try to reach zero bugs.  Personally, I think it's your obligation to do your best to find and kill bugs before your customers do -- if you want to keep your customers and your business.

What to do?  Shall we developers simply pretend bugs don't exist and keep on creating new releases?  Certainly not.  Patches to the rescue.

When you haven't yet proven that you can release software with little to no defects, you are obligated to follow the patch principle.  Your customers are counting on it.  The idea is simple.  After you release a new version of your software (major or minor release -- any version that introduces new or changed functionality) you promise to quickly and frequently release patches for the bugs that are reported by your customers.  You must not wait until the next release because guess what...there will be more bugs in that release too. (To think that 'the next release will not contain bugs' is to misunderstand developers and software development.) You should not wait and batch them up either and you should not endlessly test the patches for weeks or even days. You create and release them -- quickly -- as your customers are broken until they've applied that patch.  They are at your mercy.  It's up to you to establish a solid, reliable patch process your customers can count on.

So, as we move into the future, I intend on pushing this same philosophy with the two open-source software projects that I'm involved with. Both of these new management systems (Grassroots and ****) will not suffer from the patchy philosophy syndrome.

When we launched our Queen Creek campus at the Queen Creek High School, we decided not to rely on the local network there.  Having learned our network reliability lessons when we ran a test church at Campo Verde High School the year before, we instead decided to put some trust into the 3G cellular network. So far it's been pretty solid.

Built not for looks but for quick setup, tear-down, and storage, this 'kiosk on a cart' solution was somewhat based on Daniel's "network in a box" configuration and our custom check-in Arena module.  The cart includes a self contained network using a used Cisco 800 series router with a 3G card that creates a VPN tunnel to our main campus.  If IT and Networking are your gig, you can read all the details about those parts over on Derek's blog.

If you're wondering if it's fast or even usable... check out this video of me doing check-in of some test children during a live, normal weekend.

Every time I see the labels print out with this setup, I'm amazed.  The browser on the kiosk is talking over the 3G VPN network, to our web server which is then sending a local print job to a printer called QC1 (or QC2), which in turn is defined to talk to the printer's print-server on a particular static IP back over the 3G VPN network.  Back and fourth over the 3G network without a hitch.

One more important thing to note. Notice how fast we can punch in a phone number? Unless you're using iPads or some other non-mouse/keyboard device, you might need to change your registry's DoubleClickSpeed setting. We've set ours to 50. That setting defines how clicks are interpreted by the OS, so if the number is too high the OS won't interpret two very fast clicks (screen touches) as two single clicks.

 
Here are a few of the things swirling around in my head that make it hard to sleep...

• thinking about new friends I met in North Africa that live in tents (refugees) and have lost everything due to the revolution
  
• having so much side work I have to turn customers away
• emails I haven't responded to
  
• still top-secret and not secret side projects such as Grassroots
• the need to do more RefreshCache 2011 prep work
• thinking about how to keep up with news, software tech, tweets, feeds, and friends
  
• trying to figure out when we can upgrade our Arena to a newer version
• potential security holes
• bugs
• bed bugs (not that I have any -- just knowing that they exist somewhere out there)
• Quantative Easing 3
• thinking about and publishing a list of things that are keeping me awake at night
• recursion

I admit defeat. I can't keep up with all the community posts in the Arena community forums anymore.  Actually, I would have to say it started at some point during 2010.  I just would not admit it until today.  I guess this might also mean the.Community has become a pretty successful tool for certain things ArenaChMS!

Anyhow as a result, I'm finally changing my forum's subscriptions -- unsubscribing from about 90% of them.  I just need to keep the most essential ones... and I'm lucky I went in there.  I also just discovered several new forums for the community developers, namely the Developer Alerts forum. Perhaps that forum will be used to announce when a Arena deprecates a method from their class library API...

My good friend and co-worker, Jason Offutt, recently started blogging over at JsonData.tumblr.com.  (Yes, not only is he a developer and an artist, he's also a great writer!)  In our spare time, Jason and I have been working on a couple of open source projects. One of them, his baby, is an MVC web application for allowing organizations to start giving campaigns w/sponsors and the other is still very young and undisclosed. The topic of open source is a frequent discussion among the guys involved in these projects and Jason just wrote a great article about open source development in the .NET world --- which is something many non-MS developers may not realize is happening.

As I mentioned at RefreshCache, it's a shame when can't utilize the tools and experience we've gained during our regular job when we continue the work of our ministry in the off hours; whether we're helping build a website for an orphanage in Nepal or writing software for a parachurch ministry.  We love Arena ChMS as application development framework, but it's not open source and we just can't use it for our other projects.

We can't control the choices of our vendors, but we can control our own actions. This is part of the reason why Jason and I are involved in these other two projects.  I'll be posting more about all this early next year.  Happy Thanksgiving!

Yesterday I used the "hide" <friend> feature on Facebook and I'm torn over it.

For me, the Facebook trend had become something I dreaded because of a particular person I know who doesn't seem to have a day job other than to FB troll every day - all day. You know the type of person I'm talking about. Not the regular person who occasionally drops an agenda-based-topic into their socially acceptable narcissistic stream of consciousness. I'm talking about the super activist whose only job is to dredge up nonsense, pre-spun statistics, so-called facts and bogus "news" which they feel is going to influence the overwhelmingly silent majority of Americans.  In order to see what my other friends were up to, I'd have to wade through 6-12 other daily, inflammatory, repulsing posts from you-know-who, and after a year or so of doing this Facebook simply sickened me.

The act of FB hiding is something I've been struggling with internally for months now.  Most the people I've talked with say, 'yeah, it's fine to silence your friend, and you need to do it.'  But I can't help but feel: what's the point in calling someone a friend if you can't stomach what they have to say.  Hiding seems like a cop out - a lie to yourself.

Coincidentally, my pastor said today, 'to outright reject every detail of what someone says without measuring its truthfulness is foolish and you'll be seen as an idiot.'  While this applies to to all parties in a debate, what if only one side is being intellectually honest and reasonable?  I suppose it's only reasonable to end that debate and walk away from the unhealthy conversation.

I'm going to try it for a while and see how it goes. At least with hiding there is still some ability to participate in each others discussions, but I will definitely be mulling my decision over and may just pull the plug on the 'friendship'.

I knew something like this could happen.  It's one of the reasons why I've always been overly thoughtful about who I accept friend requests from. I don't generally think of my FB friends as simply co-workers or acquaintances.  I consider them real friends (someone I could call on for help) and I want to keep it that way.

[Note: In yet another coincidence, shortly after writing this post, Facebook announced a new version of "Groups" which could be the answer to the problem I've just described -- provided the trolls take their constant activism to their activist groups.]