Last night I spoke to Orange County SQL user group on SQL Azure, Microsoft’s cloud based relational database. SQL Azure, dubbed as re-launch of SQL Data Services (SDS) (and later SQL Services) is a cloud-based service from Microsoft offering data storage capabilities for Azure Services Platform. In the talk I discussed the challenges of putting a relational database in the cloud and Microsoft’s adaption to user feedback that they wanted SQL server in the cloud and not a schema-less entity-attribute value tables as offered by other vendors and previously was the main focus of SDS. Azure team changed course for a better customer targeted implementation catering to the relational cloud even though conventional wisdom states that relational databases aren’t as scalable as the schemaless Entity-Attribute-Value (EAV) tables used by Amazon SimpleDB, and Google’s BigTable and the App Engine’s data store.

Slides SQL Azure Presentation.pptx and code sample Connectivity.rar

Not surprisingly, a lot of focus during these talks are put on short-comings of the SQL Azure instead of the wonderful capabilities it offer. Being a VLDB aficionado, I found the attempt of putting the relational database in the cloud quite heroic to be honest. Like any v1 product, there are certain limitations but again IMHO, community needs to understand the large scale distributed database implementation issues and therefore comparing it against SQL 2008 on premise is not really an apple to apple comparison.

Along with slides and code samples, I demonstrated the connectivity from SSMS (workaround for connecting with SQL Server Management Studio), discussed SQL Azure Pricing and SQL Azure SLA (Service Level Agreement) for which “Monthly Availability” of 99.9% with a 9.99 fee, is pretty awesome.

• Web Edition:  Up to 1 GB relational database = $9.99 / month
• Business Edition:  Up to 10 GB relational database = $99.99 / month
• Bandwidth = $0.10 in / $0.15 out / GB

The new and improved control panel allows to do simple tasks such as firewall IP additions, connection strings and database provisioning quite easy.

When connecting to SQL Azure, please ensure that your firewall settings (as shown below) includes the IP range you are trying to connect from.

Feedback from Attendees (in the order of strong opinions)

·        Connectivity with SSMS needs to be fixed ASAP to avoid the workarounds

·        10GB is too small and makes the v1 not enterprise ready.

·        Error log un-availability in V1 could be a deal breaker for some DBA’s.

·        SQL Agent Support is an absolute must have from a DR and on-premise replica standpoint.

·        Programmatic way of finding the size of database

·        Linked Server Support is really important.

·        CLR Support is needed for a lot of practical purposes.

·        SQL Profiler support should be there.

The slides, code samples and links from the talk are as follows.

·        SQL Azure Home

·        SQL Azure, Let’s get started - Lynn Langit

·        SQL Azure Training Kit

·        SQL Azure Explorer Add-in

·        SQL Azure Migration Wizard v1.9

·        SQL Azure Explorer on Channel 9

·        David Yack on Starting with SQL Azure

·        geekSpeak: REST and the Windows Azure Services Platform with Adnan Masood

·        Getting Started with SQL Azure | David Gristwood | Channel 9

·        Hands on with SQL Azure (CTP 2) | David Gristwood | Channel 9

·        Billing system testing behind Microsoft's SQL Azure outage this week

·        Amazon Attempts to Preempt PDC 2009 Release of SQL Azure with MySQL 5.1 Relational Database Service

·        SQL Data Services Abandons REST for TDS API and Knocks My Socks Off

·        Frequently asked Questions about SQL Azure

·        Stephen Forte`s Blog - Building a RESTful application with SQL Azure

·        SQL Pass Summit on Azure

·        Project Rivera, Windows Azure Code samples

PS. Conveniently Amazon attempts to Preempt PDC 2009 Release of SQL Azure with MySQL 5.1 Relational Database Service Amazon Relational Database Service (Amazon RDS) Beta, announced on 10/27/2009, which delivers pre-configured MySQL 5.1 instances with up to 68 GB of memory and 26 ECUs (8 virtual cores with 3.25 ECUs each) servicing up to 1 TB of data storage

If I had a dollar for every time someone asked me this question, I would have enough money to buy Trevor Hastie's The Elements of Statistical Learning Second Edition :). Anyways, here is a good explanation from Algorithm's of the intelligent web on what is so naïve about naïve Bayesian?

"This is the calculation of the conditional probabilities p(Y|X). The term naïve has its origin in this method. Note that we’re seeking the probability of occurrence for a particular instance, given a particular concept. But each instance is uniquely determined by the unique values of its attributes. The conditional probability of the instance is, in essence, the joint probability of all the attribute value conditional probabilities. Each attribute value conditional probability is given by the term (aV.getCount()/concept-Priors.get(c)). In the preceding implementation, it’s assumed that all these attribute values are statistically independent, so the joint probability is simply the product of the individual probabilities for each attribute value. That’s the “naïve” part. In general, without the statistical independence of the attributes, the joint probability wouldn’t be equal to that product."

And the interesting part is

"We use quotes around the word naïve because it turns out that the naïve Bayes algorithm is very robust and widely applicable, even in problems where the attribute independence assumption is clearly violated. In fact, it can be shown that the naïve Bayes algorithm is optimal in the exact opposite case—when there’s a completely deterministic dependency among the attributes"

BTW, Algorithm's of the intelligent web is this excellent book by Haralambos Marmanis and Babenko Dmitry; recommended reading.

Tomorrow I will be speaking to the Southbay.NET users group in Torrance. Following is the abstract of my talk. For details and directions, please see the link below.

Web Application Testing using WATIN
Thursday, October 8th, 2009 at 6:30 PM

A while ago I did a blog post on WCF-ASMX interoperability which came out from an experience I had in service collaboration with a partner. In a recent conversation with Jeff Bergman, a friend and co-worker, there was a question of WCF or not to WCF on a project which requires similar service interop and since I believe in Atwood’s theory of strong opinions held weekly, I jumped in with the opinion of using WCF contrary to my own earlier recommendation.

Jeff of course disagreed and pointed out the Christian’s Weyer’s article on flattening. For my excellent logging and tracing in WCF argument, he stated that “for asmx, you can add the [SoapLoggerExtensionAttribute] to get some kind of logging of soap requests”. Therefore I am quoting his argument below; I am still somewhat unconvinced 'in principle' but pragmatically speaking, he drives a hard bargain!

“At the end of the day, I think the WCF architecture is powerful but with power comes complexity, complexity in configuration files and complexity in the wsdl it generates and complexity in the pipeline architecture and how you can plug into it.

As for Rest, the WCF team has developed the MVC Rest SDK.

I prefer convention over configuration and simplicity of code instead of injection as a general principle.

My preference would to be to use WCF in situations where you have more control over the client (internal uses mainly) and want to support different communication channels.  At the end of the day when dealing with partners, Interoperability is the biggest challenge and making things as simple to consume as possible is desirable.”

Have you encountered an interop issue with your WCF service and what has been your approach? Leave a comment or email me.

Happy Coding!

The Large Hadron Collider was created to help unlock the secrets of the universe. And also to create a working SOA implementation. (SOAFacts.com)

In a recent all-things-technology dinner conversation with a friend and developer extraordinaire Rashid, we talked about his recent purchase, Thomas Erl’s Service-oriented architecture: concepts, technology, and design. During this discussion he rightfully pointed out how virtually any and every distributed system is being dubbed “service oriented” raising a great point that since a lot has been said about “what SOA is?”, we should now talk about “what SOA is NOT” since this might be a able to clear things up as compared to, if it walks like SOA and talks like SOA... Hence I figured a typical non-SOA conversation may go as follows.

RealWorldPHB: “Do we have service oriented architecture? I have been hearing good things about it lately and the new partners really prefer it. Let’s go and buy one SOA or do they come in bulk?”

PerpetuallyPleasingIncompetentArchitect: “Of course we do have SOA sir, we are using web services for a long time, ahead of the curve!”

RWPHB: “Great; so we promote reuse, share contracts and schemas, do all those good things?”

PPIA: “No no, that’s too much work. Instead we share the dll’s and email it to our partners as soon as we sign the contract so they can build according to the specs. You gotta have specs for SOA”

RWPHB: “Even better; who cares about interop, right? If they have their SOA in order, they should be able to talk to us!”

PPIA: “Yessir!, you are right. We are loyal to our platform and we try to enforce uniformity, its better this way. Also our services share one database to ensure concurrency and integrity; someone called it violation of autonomous service principle and a bottleneck but we got rid of him!. We also bought that expensive SOA hardware solution and matching software suite to make sure our SOA works fine”.

RWPHB: “Glad you got rid of darn naysayer, these people upset me!. Anyways, so when we release the API update next month, how is going to work in SOA?”

PPIA: “Well, we want everyone to strictly follow our standards so we made sure our SOA does not allow any versioning. All 2000+ partners will be informed on deployment night to update their interfaces or they won’t be able to use our services. Now this is iron SOA as in iron fist!. I should patent this term.”

RWPHB: “Not sure what that means but it sounds good! I am going to tell board of directors we have SOA, woot!”

Now I should mention as a disclaimer that above mentioned conversation is a poor satirical attempt on my part on general state of SOA. Any similarity to actual persons or SOA’s, living or dead is purely coincidental. And that if you hear any such conversations in the corridors of power, hide! (No not really, educate please).

To make sure that you don’t have one of these conversations, let’s design with SOA tenants in mind. If this sounds too fancy, a simple breakdown of what is NOT SOA would be as follows.

• Boundaries are Not explicit
• Services are Not Autonomous
• Services do Not share Schema and Contract, but Class
• Compatibility is Not based upon Policy

There you go! This sums up all the things which would make your SOA design, a non-SOA design.

Ok, may be just adding a negation operator is not the best approach, let’s go with catch phrase methodology.

You might be a non-SOA if

• Your services are too much dependent on each other.
• Your service does not support versioning in contracts.
• You have to pass around jars and dll’s to share contracts.
• If your contracts and implementation are not properly isolated for consumers.
• You have an enterprise SOA strategy even though your business stakeholders didn’t sit down with IT architects to examine business processes across the organization.
• Your services use SOAP as a panacea when interop could have been better achieved with RESTful services.
• Data is ignored and governance means a config file to turn off the service response to 500.
• Your entire SOA strategy is implementing web services / BPM  / CEM / any other TLA including SOA (this will make it a circular definition).
• A client is passing a database connection string to your service call…

Yeah, I agree the last one is probably just plain bad design.

and as if this has not been fun enough, let’s end with a knock-knock joke courtesy of SOAFacts.

Knock, knock
Who's there?
SOA
SOA who?
You're fired.

References

• What SOA is not
• Ten ways to tell it's not SOA
• What you don't want to know about SOA
• SOA Facts (Humor)

• Azure Presentation Slides
• Azure Hands On Lab PDF
• Azure Hands On Lab Source

Following is the list of WCF out-of-the-box bindings, brief information about them (from MSDN) and the link which points to further details. I have ordered them according to the frequency of use in my opinion, YMMV.

BasicHTTPBinding
Represents a binding that a Windows Communication Foundation (WCF) service can use to configure and expose endpoints that are able to communicate with ASMX-based Web services and clients and other services that conform to the WS-I Basic Profile 1.1. The BasicHttpBinding uses HTTP as the transport for sending SOAP 1.1 messages. A service can use this binding to expose endpoints that conform to WS-I BP 1.1, such as those that ASMX clients consume. Similarly, a client can use the BasicHttpBinding to communicate with services exposing endpoints that conform to WS-I BP 1.1, such as ASMX Web services or services configured with the BasicHttpBinding. Security is turned off by default, but can be added setting the mode attribute of the <security> of <basicHttpBinding> child element to a value other than None. It uses a "Text" message encoding and UTF-8 text encoding by default.

webHTTPBinding
A binding used to configure endpoints for Windows Communication Foundation (WCF) Web services that are exposed through HTTP requests instead of SOAP messages.The WCF Web Programming Model allows developers to expose WCF Web services through HTTP requests that use "plain old XML" (POX) style messaging instead of SOAP-based messaging. For clients to communicate with a service using HTTP requests, an endpoint of the service must be configured with the WebHttpBinding that has the WebHttpBehavior attached to it. The WCF Web Programming Model also requires that the individual service operations are annotated with the WebGetAttribute or WebInvokeAttribute attributes. This defines a mapping from a URI and HTTP method to the service operation, as well as the format of the messages used to call the operation and return the results. Support in WCF for syndication and ASP.AJAX integration are both built on top of the WCF Web Programming Model.

wsHttpBinding
Represents an interoperable binding that supports distributed transactions and secure, reliable sessions. The WSHttpBinding is similar to the BasicHttpBinding but provides more Web service features. It uses the HTTP transport and provides message security, as does BasicHttpBinding, but it also provides transactions, reliable messaging, and WS-Addressing, either enabled by default or available through a single control setting.

netTCPBinding
A secure, reliable binding suitable for cross-machine communication. The NetTcpBinding generates a run-time communication stack by default, which uses transport security, TCP for message delivery, and a binary message encoding. This binding is an appropriate Windows Communication Foundation (WCF) system-provided choice for communicating over an Intranet.

BasicHTTPContextBinding
Enables context for the BasicHttpBinding to be exchanged with HTTP cookies as the context exchange mechanism. The BasicHttpContextBinding adds a ContextBindingElement to the stack of BindingElement objects in the system-provided BasicHttpBinding. This enables SOAP headers to be used to exchange context.

NetMsmqBinding
Represents a queued binding that is suitable for cross-machine communication.
The NetMsmqBinding binding provides support for queuing by leveraging Microsoft Message Queuing (MSMQ) as a transport and enables support for loosely coupled applications, failure isolation, load leveling and disconnected operations.

wsDualHttpBinding
A secure and interoperable binding that is designed for use with duplex service contracts that allows both services and clients to send and receive messages.The WSDualHttpBinding provides the same support for Web Service protocols as the WSHttpBinding, but for use with duplex contracts. WSDualHttpBinding only supports SOAP security and requires reliable messaging. This binding requires that the client has a public URI that provides a callback endpoint for the service. This is provided by the ClientBaseAddress. A dual binding exposes the IP address of the client to the service. The client should use security to ensure that it only connects to services it trusts.

netPeerTCPBinding
Provides a secure binding for peer-to-peer network applications.The NetPeerTcpBinding binding provides support for the creation of peer networking applications that use a TCP-level peer-to-peer mesh infrastructure.

netNamedPipeBinding
Provides a secure and reliable binding that is optimized for on-machine communication.
The NetNamedPipeBinding generates a run-time communication stack by default, which uses transport security, named pipes for message delivery, and a binary message encoding. This binding is an appropriate Windows Communication Foundation (WCF) system-provided choice for on-machine communication. It also supports transactions.

MSMQIntegrationBinding
The MsmqIntegrationBinding class maps Microsoft Message Queuing (MSMQ) messages to Windows Communication Foundation (WCF) messages.

netTCPContextBinding
Provides a context-enabled binding for the NetTcpBinding. The NetTcpContextBinding adds a ContextBindingElement to the stack of BindingElement objects in the system-provided NetTcpBinding. This enables SOAP headers to be used to exchange context.

ws2007HttpBinding
Represents an interoperable binding that derives from WSHttpBinding and provides support for the updated versions of the Security, ReliableSession, and TransactionFlow binding elements. The WS2007HttpBinding class adds a system-provided binding similar to WSHttpBinding but uses the Organization for the Advancement of Structured Information Standards (OASIS) standard versions of the ReliableSession, Security, and TransactionFlow protocols. No changes to the object model or default settings are required when using this binding.

wsHttpContextBinding
The WSHttpContextBinding adds a ContextBindingElement to the stack of BindingElement objects in the system-provided WsHttpBinding. This enables SOAP headers to be used to exchange context when HTTP cookies are not enabled.

wsFederationHttpBinding
A secure and interoperable binding that supports federated security.

• Automated Service Management
• Scalable Hosting
• Manage Services, not just servers
• Service Model and Code
• High Availability
  
• Rich Developer Experience (test it from ur local machine using Visual Studio)

• Generative URIs are handled by including support for parameterization of URI components.
• The base set of HTTP methods (GET, POST, PUT, DELETE, HEAD) are specifically supported by the WADL schema but the method enumeration is open to use with other methods such as those specified by WebDAV.
• Representation parameters are hints to processors that point out interesting parts of a resource representation. As such they may be used or ignored as desired. I think they'll be useful for RPC-like interactions but less so for more document oriented work.
• Sibling representation elements represent alternative representations of the same resource. This means you can describe a resource that is offered in alternate formats, e.g. XML or HTML.
• The design center is XML/HTTP but this doesn't preclude use with alternate representation formats: the representation/@mediaType attribute provides the necessary hook.

KDD 2008 was a great learning experience, providing opportunities for life-long learning, career development, and professional networking. It helps very much in getting to know the data mining community, the current trends in knowledge discovery & machine learning areas and that all these prolific authors and researchers are actually humans, not robots like previously thought.

Social Networking was the dominating theme of the conference and research areas specified, no doubt about that. They key sessions were as follows.

• Trevor Hastie of Stanford University on “Regularization Paths and Coordinate Descent
• Thore Graepel of Microsoft Research on “Large Scale Data Analysis and Modeling in Online Services and Advertising”
• Michael Schwarz of Yahoo! Research on “Internet Advertising and Optimal Auction Design”
• Jitendra Malik of the University of California Berkeley on “The Future of Image Search"

One of my personal favorites was Foster Provost and Jennifer Neville's tutorial session on predictive modeling in social networks. Also, I got a chance to meet and talk to the the following luminaries of the genre.

• Professor Foster Provost, editor-in-Chief of the journal Machine Learning
•  Ron Kohavi (GM for Microsoft's Experimentation Platform,)
• Gregory Piatetsky-Shapiro (Chair ACM SIGKDD)

And to see the following

• Christos Faloutsos
• Usama Fayyad
• Trevor Hastie
• Raghu Ramakrishnan (author of cow book)

However, I missed the chance of meeting Dr. Jaiwei Han. He had to leave early.

It was a well organized event with breaks. There are a few suggestions I have for improvement.

1. Provide a voting mechanism to allow people to choose their sessions of liking in advance; allocate the size of room according to the interest. This might not be perfect but will provide a good estimation. This is because some of the rooms were completely packed when people were sitting on the floor in the alleyway and some of them were half empty.

2. Full disclosure and reproducibility is important in academia and research. Some of the data used in the papers and presentations was unavailable for verification of the claims due to the proprietary nature of it, especially some of the vendor specific presentations (Yahoo, Microsoft and Orkut etc). There are very effective anonymization and privacy preserving techniques to allow the sharing.

3. Slides of the presentations should be made available to the attendees.

and next time, J'aime la vie en Paris!

Day 3 started with Invited Talk of Dr. Michael Schwarz from Yahoo! Research on “Internet Advertising and Optimal Auction Design” who discussed Generalized English Auctions  and Internet advertising, generalized second price options. (Details can be read on Akshay Java’s Social Media research blog here). It was an interesting talk pertaining to how almost every transaction follows the modern auction model and what approaches can be used to maximize the throughput and ROI.
Later during regular sessions, I attended the Discovery and Detection research session which was focused on outlier analysis. It comprised of the following presentations.
25-minute presentations

• Automatic Identification of Quasi-Experimental Designs for Discovering Causal Knowledge. D. D. Jensen, A. S. Fast, B. J. Taylor, M. E. Maier.
• Discrimination-aware Data Mining. D. Pedreschi, S. Ruggieri, F. Turini.

15-minute presentations

• Local Peculiarity Factor and Its Application in Outlier Detection. J. Yang, N. Zhong, Y. Yao, J. Wang.
• Angle-Based Outlier Detection in High-dimensional Data. H. Kriegel, M. Schubert, A. Zimek.
• Anomaly Pattern Detection in Categorical Datasets. K. Das, J. Schneider, D. B. Neill.

Lunch was sponsored by Yahoo! and it was pretty cool décor with their gadgets, puzzles and YahooDokus.  Later in the afternoon I attended
25-minute presentations

• Factorization Meets the Neighborhood: a Multifaceted Collaborative Filtering Model. Y. Koren.
• Combinational Collaborative Filtering for Personalized Community Recommendation. W. Chen, D. Zhang, E. Y. Chang.

And then

• (R) Scaling Up Text Classification for Large File Systems. G. Forman, S. Rajaram.
• (I) Data Mining Using High Performance Data Clouds: Experimental Studies Using Sector and Sphere. R. Grossman, Y. Gu Short.

I later got a chance to talk to Dr. Grossman about the cloud computing initiative that he is very passionate about. He discussed in his presentation how Sector is approximately twice as fast as Hadoop and how Sector has been used to distribute the Sloan Digital Sky Survey (SDSS) via the web site sdss.ncdm.uic.edu.
The day concluded with Poster reception where I get to talk to several authors and presenters including Wen-Yen Chen, Pooja Mittal, Yabo-Arber Xu, Kaustav Das of Anomaly Pattern Detection in Categorical Datasets and one of the authors of Information Extraction from Wikipedia, not sure who.

Day 4

The last day of conference started with Jitendra Malik’s invited talk on “The Future of Image Search”.  (Greg Linden’s Blog Post about the talk). It was a great talk where Jitendra discussed the evolution of vision, image search, shortcomings of tagging and textual taxonomies and pushed for "category recognition" for objects in images.
Later there were the following excellent sessions.

• Context-Aware Query Suggestion by Mining Click-Through and Session Data. H. Cao, D. Jiang, J. Pei, Q. He, Z. Liao, E. Chen, H. Li. (Best Application Paper Award Winner)
• Scalable and Near Real-Time Burst Detection from eCommerce Queries. N. Parikh, N. Sundaresan.
• Using Predictive Analysis to Improve Invoice-to-Cash Collection. S. Zeng, P. Melville, C. A. Lang, I. Boier-Martin, C. Murphy.
• TagMark: Reliable Estimations of RFID Tags for Business Processes. L. W. F. Chaves, E. Buchmann, K. Böhm.

The conference concluded with closing remarks from the general chair, Ying Li.

Day 2 started with Trever Hastie’s talk on regularization paths and coordinate descent.  It was great to see Dr. Hastie passionately speaking about the coordinate descent, logistic regression and fitting.  The keynote’s topic was “Regularization Paths and Coordinate Descent” and following is a brief abstract from the talk.

"In a statistical world faced with an explosion of data, regularization has become an important ingredient. In many problems, we have many more variables than observations, and the lasso penalty and its hybrids have become increasingly useful. This talk presents some effective algorithms based on coordinate descent for fitting large scale regularization paths for a variety of problems. Joint work with Rob Tibshirani and Jerome Friedman"

After the keynote talk, there were combined and research sessions. I attended one with social Networks which comprised of the following presentations.
25-minute presentations

• The Structure of Information Pathways in a Social Communication Network. G. Kossinets, J. Kleinberg, D. Watts.
• Influence and Correlation in Social Networks. A. Anagnostopoulos, R. Kumar, M. Mahdian.
• Weighted Graphs and Disconnected Components. M. McGlohon, L. Akoglu, C. Faloutsos.

15-minute presentations

• Microscopic Evolution of Social Networks. J. Leskovec, L. Backstrom, R. Kumar, A. Tomkins.
• Mobile Call Graphs: Beyond Power-Law and Lognormal Distributions. M. Seshadri, S. Machiraju, A. Sridharan, J. Bolot, C. Faloutsos, J. Leskovec.
• Feedback Effects between Similarity and Social Influence in Online Communities. D. Crandall, D. Cosley, D. Huttenlocher, J. Kleinberg, S. Suri.

During lunch which was sponsored by Microsoft adCenter labs, they talked about challenges in advertising and applying it to get the best out of revenue share and context base hits. They also announced adCenter labs challenge which I have yet to find any information about online.
Later in the afternoon, Microsoft Research’s Thore Graepel talked about Large Scale Data Analysis and Modeling in Online Services and Advertising. It was a very interesting and pragmatic presentation about the real world problems in online search and advertising. Even though the first part of presentation was online ranking and matchmaking heavy, the later discussion on advertising made up for it.
Then there were 25-minute presentations

• ArnetMiner: Extraction and Mining of Academic Social Networks. J. Tang, J. Zhang, L. Yao, J. Li, L. Zhang, Z. Su.
• Identifying Authoritative Actors in Question-Answering Forums - The Case of Yahoo! Answers. M. Bouguessa, B. Dumoulin, S. Wang. (the paper was on Belinko’s website but not there anymore, interesting?)

And on a separate track,
15-minute presentations

• Automated Cyclone Discovery and Tracking using Knowledge Sharing in Multiple Heterogeneous Satellite Data. S.-S. Ho, A. Talukder.
• Spotting Out Emerging Artists Using Geo-Aware Analysis of P2P Query Strings. N. Koenigstein, Y. Shavitt, T. Tankel.
• Land Cover Change Detection: A Case Study. S. Boriah, V. Kumar, M. Steinbach, C. Potter, S. Klooster.

Later after the automated cyclone discovery session, I got a chance to meet Dr. Talukder of Jet Propulsion Laboratory/NASA to talk about a mutual acquaintance, Dr. Homayoun Seraji

This concluded day 2 of the conference.

The MSDN webcast on geekSpeak: REST and Windows Communication Foundation 3.5 went very well; Since REST is a very broad topic and there were tons of questions, I didn’t get a chance to show all the demos however the sample code can be downloaded from here.

Sample Code

Also, keep an eye on geek speak blog for future updates. Overall. there is a lot of concern about security in REST. I’ll be doing a series of blog posts on security in REST in near future however in the mean time, following resources would be provide a good starting point.

Mark O'Neill's Radio Weblog
Message Level Security in REST

Taking Amazon S3 as a model for secure REST services can be one way to implement security in REST. As mentioned in this article by Eric Heuveneers

“Amazon S3 REST resources are secure. This is important not just for your own purposes, but also because customers are billed depending on how their S3 buckets and objects are used. An AWSSecretKey is assigned to each AWS customer, and this key is identified by an AWSAccessKeyID. The key must be kept secret and will be used to digitally sign REST requests. S3 security features are:

• Authentication: Requests include AWSAccessKeyID
• Authorization: Access Control List (ACL) could be applied to each resource
• Integrity: Requests are digitally signed with AWSSecretKey
• Confidentiality: S3 is available through both HTTP and HTTPS
• Non repudiation: Requests are time stamped (with integrity, it's a proof of transaction)

The signing algorithm is HMAC/SHA1 (Hashing for Message Authentication with SHA1).’

Reference: Introduction to Amazon S3 with Java and REST

Links to the books and reference articles mentioned in the webcast are as follows. Please feel free  to send me your questions and comments on my email 

Books

• RESTFul .NET
• RESTFul Web Services
  
  References

• The Weekly Source Code 32- Atom, AtomPub and BlogSvc, an AtomPub Server in WCF
• WCF 3.5 Utilities
• WCF 3.5 RSS / ATOM  Syndication Support
• Dare Obasanjo aka Carnage4Life - ETech 2005 Trip Report Building a New Web Service at Google
• Defining REST based Formats
• InfoQ Dan Diephouse on Atom, AtomPub, REST and Web Services
• MIX07 Buzzcast #12 - Steve Maine - Navigating the Programmable Web MIX07 Buzzcast Channel 9
• Scott Hanselman's Computer Zen - Web Services
• Showcase of Live ASP.NET MVC Sites - Mike Bosch's Blog on .NET
• The Highs and Lows of REST
• TRUVEO - Complete ASP.NET sample REST Service Calls
• WCF & REST at MIX08 The Tale of MySpace APIs
• WCF Web Programming Model Overview
• WebServiceStudio - Home (CodePlex Project)
• YABE Yet Another Blogging Engine - Home (CodePlex Project)
• WADL (CodePlex Project)

KDD 2008 Day 1

• Set 1
• Set 2
• Set 3
• Set 4

KDD 2008 Day 2

• Set 1
• Set 2
• Set 3
• Set 4

KDD 2008 Day 3

• Set 1
• Set 2
• Set 3
• Set 4
• Set 5

KDD 2008 Day 4

• Set 1
• Set 2

ACM's KDD 2008 is the annual premier international forum for data mining researchers and practitioners from academia, industry, and government to share their ideas, research results and experiences. This year this event was held in Loews Lake Las Vegas resort where Jeff Bergman and I attended it. Details of the program can be found here http://www.kdd2008.com/program.html and the summary is as follows.

Full Day Workshop W1 - ADKDD'08
Full Day Workshop W2 - WEBKDD'08
Full Day Workshop W3 - Sensor-KDD
Full Day Workshop W4 - PinKDD'08
Full Day Workshop W5 - SNA-KDD
Full Day Workshop W13 - Multimedia Data Mining

9:00 am - 12:00 pm
Half Day Workshop W6 - KDD CUP and Mining Medical data
Half Day Workshop W7 - Multiple Information Sources
Half Day Workshop W11 - BIOKDD08
Half Day Workshop W12 - Mining for Business Applications

9:00 am - 12:00 pm
Tutorial - Mining Massive RFID, Trajectory, and Traffic Data Sets
Tutorial - Predictive Modeling with Social Networks
Tutorial - Mining Uncertain and Probabilistic Data: Problems, Challenges, Methods, and Applications
Tutorial - Detecting Clusters in Moderate-to-High Dimensional Data: Subspace Clustering, Pattern-based Clustering, and Correlation Clustering

2:00 pm - 5:30 pm Half Day Workshop
W8 - Large Scale Recommender Systems and NetFlix Prize
W10 - Mining using Matrices and Tensors

2:00 pm - 5:00 pm
Tutorial - Blogosphere: Research Issues, Applications, and Tools
Tutorial - Graph Mining and Graph Kernels
Tutorial - Applied Text Mining

6:15 pm - 6:45 pm : Award Presentations

6:45 pm - 7:30 pm : Innovation Award Talk

Day 1 was very informative and provided good learning experience. The program included several full day workshops and tutorials listed below.

·         J. Han, J. Lee, H. Gonzalez, X. Li, "Mining Massive RFID, Trajectory, and Traffic Data Sets"
Jiawei Han, Jae-Gil Lee, Hector Gonzalez, Xiaolei Li
Department of Computer Science, University of Illinois at Urbana-Champaign

·         J. Neville, F. Provost, "Predictive Modeling with Social Networks"
Jennifer Neville, Purdue University
Foster Provost, New York University

·         J. Pei, M. Hua, Y. Tao, X. Lin, "Mining Uncertain and Probabilistic Data: problems, Challenges, Methods, and Applications"
Jian Pei, Simon Fraser University, Canada
Ming Hua, Simon Fraser University, Canada
Yufei Tao, The Chinese University of Hong Kong
Xuemin Lin, The University of New South Wales, Australia

·         H. Kriegel, P. Kroger, A. Zimek, "Detecting Clusters in Moderate-to-High Dimensional Data: Subspace Clustering, Pattern-based Clustering, and Correlation Clustering"
Hans-Peter Kriegel, Peer Kröger, and Arthur Zimek
Institute for Informatics, Ludwig-Maximilians-Universitat Munchen, Germany

·         H. Liu and N. Agarwal, "Blogosphere: Research Issues, Applications, and Tools". Huan Liu, Arizona State University, Nitin Agarwal, Arizona State University
R. Feldman, L. Ungar, "Applied Text Mining"

Social Networking being the prominent theme at the conference, I decided to get a head start by attending the half day tutorial on "Predictive Modeling in Social Networks" by Jennifer Neville and Foster Provost.  The abstract from the tutorial is as follows.

Recently there has been a surge of interest in methods for analyzing complex social networks: from communication networks, to friendship networks, to professional and organizational networks. The dependencies among linked entities in the networks present an opportunity to improve inference about properties of individuals, as birds of a feather do indeed flock together. For example, when deciding how to market a product to people in MySpace or Facebook, it may be helpful to consider whether a person's friends are likely to purchase the product.

This tutorial will explore the unique opportunities and challenges for modeling social network data. We will begin with a description of the problem setting, including examples of various applications of social network mining (e.g., marketing, fraud detection). We will then present a number of characteristics of social network data that differentiate it from traditional inference and learning settings, and outline the resulting opportunities for significantly improved inference and learning. We will discuss specific techniques for capitalizing on each of the opportunities in statistical models, and outline both methodological issues and potential modeling pathologies that are unique to network data. We will give links to the recent literature to guide study, and present results demonstrating the effectiveness of the techniques.

Dr. Provost started by establishing the core foundation for social networking and further get in depth with network targeting, disjoint inference, learning & classification, wvRN, ACORA, RBC, RPT, SLR and context of collective inference. Dr. Neville then continued with Gaussian random fields and elaborated with her work on questionable broker detection. Semi-supervised learning, conventional bias vs. variance analysis, homophily, social influence, external factors and open research issues were also part of tutorial. Later in a discussion with Dr. Provost, he mentioned that the collaborative techniques described can also be implemented for outlier analysis which was encouraging.

For the second tutorial, I attended the "Graph Mining and Graph Kernel" tutorial by Karsten M. Borgwardt (http://mlg.eng.cam.ac.uk/~karsten/) and Xifeng Yan (IBM Research Center). This tutorial presented a comprehensive overview of the techniques developed in graph mining and graph kernels and examines the connection between them.  As described by authors, “The goal of this tutorial is i) to introduce newcomers to the field of graph mining, ii) to introduce people with database background to graph mining using kernel machines, iii) to introduce people with machine learning background to database-oriented graph mining, and iv) to present exciting research problems at the interface of both fields.”

Applied Text mining tutorials by Dr. Ronen Feldman & Dr. Lyle Unger was also an excellent talk. Dr. Feldman, author of applied text mining, has a great style of pragmatic discussion and connects with the audience really well. I am looking forward to his future presentation and discuss the idea of natural language corpus extraction implementations in Text mining for my Urdu machine translation work; he must have some great ideas about it.

After the tutorials Bing Liu, the program chair presented conference statistics; apart from all other numbers, salient ones are submission from the US, 323 papers out of which 81 were accepted. In total there were 593 submissions and 118 accepted ones, a less than 20% or less than 1 out of 5 ratio! These guys are picky.

Then came the best research paper award, best application paper award, student travel awards, KDD dissertation award, KDD Cup awards, KDD innovation award and finally concluded on innovation award talk by Raghu Ramakrishnan.  KDD Cup 2008 winning announcements in medical data mining was a highly practical and quite challenging problem. Details of the cup submissions can be seen here. http://www.kdd2008.com/kddcup.html

Dr. Ramakrishnan is the author of “Cow Book” and his final talk for the day covered his past research and a broad spectrum of future directions of information retrieval. With educated “predictions” from  a seasoned data miner, the first day concluded.

I’m very much looking forward to tomorrow’s sessions; till then, happy mining.

I've taken a lot of photos of the presentations Photos of the event are shared on the facebook. Click here to see them.

Testing web services is a pivotal part of contemporary enterprise project life cycle; developers, QA and even system guys do it to validate different aspects of the middleware. This testing comes with its own set of challenges; Aside from being ineffective, testing complex types is not possible via the default browser based test client and WCF services don’t even offer it!. So what’s the remedy?

This webcast demonstrates web service studio and WCF Test client, two tools specifically designed to test web services without the need of writing custom test harnesses. Web Service Studio is a codeplex project is the revival of good old .NET Webservice Studio tool. Web Service Studio is a tool to invoke webmethods interactively. The user can provide a WSDL endpoint. On clicking button Get the tool fetches the WSDL, generates .NET proxy from the WSDL and displays the list of methods available. The user can choose any method and provide the required input parameters. On clicking Invoke the SOAP request is sent to the server and the response is parsed to display the return value. My intent is to further enhance it to add the support for WCF, Nullable Types and REST style API to allow a complete composite type testing from one tool. For details on WCF Test Client, please see my article here.

Webcast on Web Service Studio and WCF Test Client

References

http://www.codeplex.com/WebserviceStudio

www.codeproject.com/KB/WCF/WCF35Utils.aspx

August and the next couple of months looks really busy. I’ll be teaching WCF at UCSD and will be attending the following conferences along with doctoral cluster meeting. Therefore I am seriously considering “The Terminal” style living.

KDD 2008, 24 – 27 Aug 2008, Loews Lake Las Vegas Las Vegas, NV
The annual ACM SIGKDD conference is the premier international forum for data mining researchers and practitioners from academia, industry, and government to share their ideas, research results and experiences. KDD-08 will feature keynote presentations, oral paper presentations, poster sessions, workshops, tutorials, panels, exhibits, demonstrations, and the KDD Cup competition.

IASA Connections, October 6 - 8, 2008, San Francisco Marriott, San Francisco, CA
I'll be speaking to IASA connections conference in San Francisco on Aspect Oriented Programming in Distributed Systems. More details here.

Microsoft PDC 2008 – 27 – 30 Oct, Los Angeles Convention Center, Los Angeles, CA
Since 1991, the Professional Developers Conference (PDC) has been Microsoft’s premier gathering of leading-edge developers and architects. Attend the PDC to understand the future of the Microsoft platform and to exchange ideas with fellow professionals. You’ll learn about upcoming products, meet Microsoft’s leaders and top engineers, write some code, and be inspired! Unplug for a few days and think about the future.  

Programming Windows Communication Foundation (WCF) (Summer 2008)
Sa, 8:00 a.m. - 5:00 p.m.
8/9/2008 - 8/23/2008
Room 134, UCSD Extension Complex, 9600 N Torrey Pines Rd, La Jolla

Programming Windows Communication Foundation (WCF) (Fall 2008)
Sa, 8:00 a.m. - 5:00 p.m.
10/4/2008 - 10/18/2008
Room 110, UCSD Extension Sorrento Mesa Center, 6925 Lusk Blvd, San Diego

OWASP’s list have been changed since 2004 in terms of priorities; XSS and inject flaws are on the rise. Details can be found on OWASP’s website.

OWASP .NET Projects
http://www.owasp.org/index.php/Category:OWASP_.NET_Project

References and Papers on Financial Data Mining

• Mine Your Way to Combat Money Laundering
• OFAC SDN List www.ustreas.gov/offices/enforcement/ofac/sdn/
• FinCen www.fincen.gov/
• FATF www.fatf-gafi.org/
• Suspicious Activity Report
• Keys to a Well Prepared Suspicious Activity Report
• A framework for data mining-based anti-money laundering research
• Profiling Behavior: The social construction of categories in the detection of financial crime; dissertation by Ana Canhoto
• Towards a Proactive Fraud Management Framework for Financial Data Streams
• T. Senator. "The financial crimes enforcement network AI system (FAIS)." AI Magazine 4, 1995.
• M. Sparrow. "The State of the Fraud Control Game; and the Impact of Electronic Claims Processing on Fraud and Fraud Control." Proceedings of the International Symposium on Criminal Justice Information Systems and Technology, 1994.
• U.S. Congress, Office of Technology Assessment (OTA). "Information Technologies for Control of Money Laundering." OTA-ITC-630. Washington, DC: U.S. Government Printing Office, September 1995.
• Zdanowicz, J.S. (2004), "Detecting money laundering and terrorist financing via data mining", Communications of the ACM, Vol. 47 No.5
• Watkins, R.C., Reynolds, K.M., Demara, R., Georgiopoulos, M., Gonzalez, A., Eaglin, R. (2003), "Tracking dirty proceeds: exploring data mining technologies as tools to investigate money laundering", Police Practice and Research, Vol. 4 No.2, pp.163-78.
• Vikram, A., Chennuru, S., Rao, H.R., Upadhyaya, S. (2004), "A solution architecture for financial institutions to handle illegal activities: a neural networks approach", Proceedings of the 37th Hawaii International Conference on System Sciences-2004
• Zhang, Z., Salerno, J.J., Yu, P.S. (2003), "Applying data mining in investigating money laundering crimes", paper presented at SIGKDD'03, Washington, DC, pp.747-52.
• Senator, T.E., Goldberg, H.G., Wooton, J. (1995), "The financial crimes enforcement network AI system (FAIS): identifying potential money laundering from reports of large cash transactions", AI Magazine, Vol. 16 No.4, pp.21-39.
• Tang, J., Yin, J. (2005), "Developing an intelligent data discriminating system of antimony laundering based on SVM", Proceedings of the Fourth International Conference on Machine Learning and Cybernetics. Guangzhou, pp.3453-7.
• Kingdon, J. (2004), "AI fights money laundering", IEEE Intelligent Systems, Vol. 5/6 pp.87
• Goldberg, H.G., Wong, R.W.H. (1998), "Restructuring transactional data for link analysis in the FinCEN AI System", Proceedings of 1998 AAAI Fall Symposium on Artificial Intelligence and Link Analysis, AAAI Press, Menlo Park, CA, .
• Fawcett, T., Provost, F. (1997), "Adaptive fraud detection", Data Mining and Knowledge Discovery, Vol. 1 No.3, pp.291-316.

• REST using WCF 3.5.pdf - Slides in PDF (529.21 KB)
• REST using WCF 3.5.pptx - Slides in Powerpoint (184.54 KB)
• RESTful Communication.docx (Excerpt from David Chappell's whitepaper)
• RestService.zip - Simple Service Source Code (3.18 KB)
• WCF3.5-SyndicationRESTSamples.zip (Uri Templates, REST-Sample MSDN Source Code) (162.61 KB)

• www.cs.uvm.edu/~icdm/10Problems/index.shtml and
• www.cs.uvm.edu/~icdm/algorithms/index.shtml

• Dependency Injection was made for ASP.NET MVC
• Inversion of Control Containers and the Dependency Injection pattern
• ASP.Net MVC Framework - Create your own IControllerFactory and use Spring.Net for DI
• TDD and Dependency Injection with ASP.NET MVC
• http://haacked.com/archive/2007/12/07/tdd-and-dependency-injection-with-asp.net-mvc.aspx
• MVC Storefront: Dependency Injection
• List of .NET Dependency Injection Containers (IOC)
• IOC and Unity compared to other containers
• Introduction to using StructureMap for Dependency Injection
• Screencast: Unity and Dependency Injection
• Dependency Injection with Spring.NET
• Context Variables in Ninject
• MVC Samples - http://www.codeplex.com/mvcsamples/

Following  are the links to my talks from the SoCal Rock and Roll Code Camp in UCSD Extension Campus, San Diego.

Using ASP.NET MVC  to build a blogging engine in 60 minutes or less.

MVC is a framework methodology that divides an application's implementation into three component roles: models, views, and controllers. ASP.NET now has built-in support for MVC style development and this session is an introduction to using this technique for building a sample application, a blogging engine.  This session will elaborate on differences between traditional ASP.NET post-back style development versus the routes and REST architecture based thinking around MVC.

• MVC Blog Engine.pptx (222.42 KB)
• YABE.06282008.zip (1.54 MB)
• www.CodePlex.com/YABE

ASPECT.NET – Aspect Oriented Programming in .NET, an Introduction

Aspect Oriented Programming (AOP) deals with factorization in code i.e. separation of common concerns, specifically cross-cutting concerns, as an advance in modularization. AOSD has been a popular trend in development for quite some time in other programming environments and IDE’s however it’s scope and exposure is limited among .NET developers.

• Aspect Oriented Programming in .NET - Final.pptx (1.09 MB)
• Aspect.NET 2.1.zip (2.19 MB)
• Safonov_Aspect_Dot_Net_2.pdf (1.08 MB)

 Collaborative Filtering 101 – An Introduction with SQL Server 2008 BI

Collaborative Filtering (CF) is defined as profiling or classification of information based on specific entity relationships i.e. making automatic predictions (filtering) about the interests of a user by collecting likelihood information from many users (collaborating). The underlying assumption of CF approach is that those who agreed in the past tend to agree again in the future. For example, a collaborative filtering or recommendation system for music tastes could make predictions about which music a user should like given a partial list of that user's tastes (likes or dislikes).

In this session, we will discuss collaborative filtering algorithms and applications in the current e-commerce systems. A wide array of topics such as market basket analysis, association trees,  singular value decomposition (SVD), naïve Bayesian classification will be briefly discussed along with the implementation of these algorithms in sites like Netflix, Amazon and digg / (google pagerank). In the second half of the talk, attendees will get to see the step by step implementation of a small scale recommender system using SQL Server 2008 business intelligence studio and C#.

• Collaborative Filtering 101.pptx (2.89 MB)
• Amazon-Recommendations.pdf (359.21 KB)
• MovieSample.zip (1.54 MB)
• ProgressPrize2007BellKorSolution.pdf (67.8 KB)
• TechEd_2003_DM_MovieClerk_App_and_Web_Service.zip (38.17 KB)
• what is netflix challenge.pdf (143.54 KB)

The UCSD Code Camp Speaker's Room; the best place to recruit speakers for your user group.

Everyone of has seen it; the default namespace TempUri reference in our otherwise neatly published WSDL, looks like an out of place not-so-thought-out part of your otherwise ironclad contract. Hence the question arises, how to get rid of it and put your company’s corresponding namespace in there.

In the asmx services, it was rather easy to do it; just modify your webservice attribute.

[WebService(Namespace = "http://tempuri.org/")]

However, in the new uncharted waters of WCF, things are wee bit more complex than this. Now you’d need to do it in multiple places. The point to remember is that an endpoint has an associated behavior with it.

The web.config file.

 Service Endpoint
<endpoint address=""                                                                                                                                           binding="basicHttpBinding"                                                                                                                                 contract="Acme.Services.WCFNamespaceSample.IService"                                                                                bindingNamespace ="http://acme.com/Acme/Services/WCFNamespaceSample/">

Behavior Endpoint
<endpoint address="mex" binding="mexHttpBinding" contract="IMetadataExchange" bindingNamespace="http://acme.com/Acme/Services/WCFNamespaceSample/"/>

Also, for the general namespace changes, you need to specify the attribute as follows.

namespace Acme.Services.WCFNamespaceSample
{
                [ServiceBehavior (Namespace = "Acme.Services.WCFNamespaceSample")]
                public class Service : IService
…

And

namespace Acme.Services.WCFNamespaceSample
{
                [ServiceContract(Namespace = "Acme.Services.WCFNamespaceSample")]
                public interface IService
…

This should do the magic. Following source code is a good place to start.

While reading up on outlier cohesion analysis in collaborative social networks, I came across this article which I found pretty interesting in the security niche; “Structural Analysis and Destabilizing Terrorist Networks” by  N. Memon et al. The article discusses key areas in network analysis such as

(i)                  cohesion analysis (such as cliques, ncliques, n-clans and k-plex) to determine familiarity, robustness and reachability.
(ii)                role analysis (such as position role index) to determine critical nodes and
(iii)               power analysis (such as degree centrality, Eigenvector centrality and dependence centrality)

This work has further been refined and published as IEEE conference on intelligence security and ADMA

Practical Algorithms for Destabilizing Terrorist Networks
N Memon, HL Larsen - Proceedings of IEEE Conference on Intelligence Security …, 2006 – Springer

And

Structural Analysis and Mathematical Methods for Destabilizing Terrorist Networks Using Investigative Data …
N Memon, HL Larsen - … Conference on Advanced Data Mining Applications (ADMA 2006), 2006 – Springer

From a generic covert network perspective, there has been previously a lot of work done for darkNet exploration, automated discovery for nodes with case studies in Allpeers, anoNet, Freenet, GNUnet, I2P , Tor, Turtle F2F and WASTE.

A good overview can be found here.

 Destabilizing dynamic covert networks
KM Carley, M Dombroski, M Tsvetovat, J Reminga, N … - Proceedings of the 8th International Command and Control …, 2003 - casos.cs.cmu.edu

The problem of network outliers is not only crucial in intrusion detection but also an interesting network theory problem where a leaf node posses attributes out of the ordinary. As discussed by N. Memon et al for the social aspect of network; “The analysis of the interaction structures that is involved in social network analysis is an important element in the analysis of the micro-macro link, the way in which individual behavior and social phenomena are connected with one another. In this perspective, social networks are both the cause of and the result of individual behavior.”

References from the paper and further readings

• http://en.wikipedia.org/wiki/Network_theory
• Social Network :: Wikipedia
• http://trackingthethreat.com/

1. Scott, J.: Social Network Analysis: A Handbook, 2 edn. Sage Publications, London 2000.

2. Wasserman, S., Faust, K.: Social Network Analysis. Cambridge University Press.1994.

3. Sageman, M.: Understanding Terrorist Networks. University of Pennsylvania Press, 2004.

4. Berry, N., Ko, T., Moy, T., Smrcka, J., Turnley, J., Wu, B.: Emergent clique formation in terrorist recruitment. The AAAI- 04 Workshop on Agent Organizations: Theory and Practice, July 25, 2004, San Jose, California, 2004. http://www.cs.uu.nl/virginia/aotp/papers.htm

5. McAndrew, D.: The structural analysis of criminal networks. In: D. Canter, L. Alison (eds.) The Social Psychology of Crime: Groups, Teams, and Networks, Offender Profiling Series, III.Aldershot, Dartmouth ,1999.

6. Davis, R.H.: Social network analysis: An aid in conspiracy investigations. FBI Law Enforcement Bulletin pp. 11–19, 1981.

7. Chen, H., Chung, W., Xu, J.J., Wang, G., Qin, Y., Chau, M.: Crime data mining: A general framework and some examples. Computer 37(4), 50–56, 2004.

8. Krebs, V.: Mapping networks of terrorist cells. Connections 24, 45–52, 2002.

9. Bonacich, P., Power and Centrality. American Journal of Sociology 92: 1170-1184, 1987.

10. Burt, R. S., Structural Holes, Cambridge, MA: Harvard University Press, 1992.

11. Hanneman, R. E., Introduction to Social Network Methods.Online Textbook Supporting Sociology 175. Riverside, CA: University of California, 200.

12. Burt, R. S., Structure, A General Purpose Network Analysis Program. Reference Manual, Newyork: Columbia University, 1990.

13. Luce, R., Perry, A.: A method of matrix analysis of group structure. Psychometrika 14, 95–116, 1949.

14. Seidman, S.B., Foster, B.L.: A graph theoretic generalization of the clique concept. Journal of Mathematical Sociology 6, 139–154, 1978.

15. Freeman, L.C.: The sociological concept of “group”: An empirical test of two models. American Journal of Sociology98, 152–166 ,1992.

16. Luce, R.: Connectivity and generalized cliques in sociometric group structure. Psychometrika 15, 169–190, 1950.

17. Mokken, R.: Cliques, clubs and clans. Quality and Quantity 13, 161–173, 1979.

18. Balasundaram, B., Butenko, S., Trukhanov, S.: Novel approaches for analyzing biological networks. Journal of Combinatorial Optimization 10, 23–39, 2005.

19. Latora, V., Massimo Marchiori How Science of Complex Networks can help in developing Strategy against Terrorism, Chaos, Solitons and Fractals 20, 69-75, 2004.

20. Memon, N. Henrik Legind Larsen, Practical Algorithms for Destabilizing Terrorist Networks, In Proceedings of IEEE Intelligence Security Conference (ISI 2006), San Diego, California, USA (to appear), 2006.

 Newman, M. E. J. The structure and function of complex networks, SIAM Review 45, 167- 256, 2003.

And on a humorous side, here is an interesting video about how NOT to do it.:)

Aspect Oriented Programming: Radical Research in Modularity

Video: Aspect-Oriented Modeling - what it is and what it's good for

Video: Anurag Mendhekar: Aspect-Oriented Programming (Dan Friedman)

PointCut Doctor
IDE Support for Understanding and Diagnosing AspectJ Pointcuts   

Aspect.NET 2.1
An aspect-oriented programming tool for Microsoft.NET

Loom.NET
The LOOM .NET project aims to investigate and promote the usage of AOP in the context of the Microsoft .NET framework.

AOSD – Annual Conference

AOSD Community website

I'll be speaking to IASA connections conference in San Francisco on Aspect Oriented Programming in Distributed Systems.

Following are further details.

Conference Page
October 6 - 8, 2008
San Francisco Marriott
San Francisco, CA

IASA02: Service Aspects—Aspect Orientated Designs in Distributed Enterprise Architecture
Adnan Masood
Aspect Oriented Programming and Aspect-Oriented software development (AOSD) support the software development paradigm which leverages separation of concerns, especially cross-cutting concerns as a next step to modularization. Separation of concerns can be defined as breaking down a program into distinct parts that overlap in functionality as little as possible. The similar concerns are factored and defined as aspects which are separated out from the main logic making the implementation more maintainable. In this session we approach the service orientation as an aspect of a distributed system. Using attribute oriented design for aspect implementation, this presentation focuses on merits of exposing service end points from business objects by using AOP practices. The attendees will: gather the understanding of AOP, a fast growing research and development area in modern software development; understand the state of affairs of AOP in the current IDE’s and programming languages especially with Spring, AspectJ and Aspect.NET; explore the rationale of aspect-based nature of services and deep dive into the open source ServiceAspect CodePlex project for a sample implementation. This session’s focus is the architecture and design practices that AOSD brings to the enterprise architecture. Best practices and design patterns followed in AOP will be discussed with a demo of Aspect.NET and ServiceAspect, which is used to publish business objects as WCF services using attributes.

List of  Speakers

List of Sessions

Returning HTTP 500 internal server error as your web service / asp.net REST responses in case of failures may  sound like an odd idea but it has its merits; with handlers and load balancing environments where you want your intelligent routing devices to redirect request to a different server based on an HTTP header response, this approach can come in handy.

In order to return an HTTP 500 server error custom response, there are two simple ways to achieve it.

By throwing an HTTP exception

throw new System.Web.HttpException(500, "Internal Server Error");

Or by modifying the headers.

       HttpContext.Current.Response.Status = "500 Internal Server Error";

       HttpContext.Current.Response.AddHeader("Status Code", "500");

       HttpContext.Current.Response.End();

Looking at the response from the HTTP headers, they are almost identical. The only difference is that due to termination of response stream in the later case, the content length is 0.

The similar result can be achieved by throwing an ApplicationException or custom exception as well because for the runtime, it still means an internal server error.

The source for the web service project can be downloaded from here. HTTP500Test-Src.zip (2.79 KB)

With the rapidly changing horizon of upcoming tools and technologies, technical certification is an organized way of studying and keeping oneself up-to-date with the latest and greatest features which software frameworks have to offer.  Even though hands on experience is an absolute necessity, usually when using a particular framework in everyday job, a developer only gets exposed to a smaller set of features pertaining to the task at hand. On the contrary, certification study provides a breadth first approach to the general underlying framework, its mechanics and features. Since it covers a much broader area, studying for a certification exam becomes a daunting task; luckily not for those equipped with tools such as uCertify’s certification study products which allow you to focus your study in the key areas and provide you with instant feedback and help during the preparation process.

uCertify’s 70-526 – C# Exam. TS Microsoft.NET Framework 2.0 Windows based client preparation tool assists you to get ready for the test in a well organized and effective way. It consists of three practice tests with 35 well diversified questions, flash cards, adaptive tests, a detailed study notes section and final exam with 40 questions. This product lets you evaluate your readiness level, provides you a test history to ensure the progress and also allow you to create a custom test, based on the areas you find yourself not so strong. The product is customizable and allow you to create your own entries for test preparation but I believe the sections such as How To’s  could have been expanded and made more effective.  A test can be conducted in both “test” and “learning mode”; detailed explanation and references are provided in the later mode for follow up and understanding. The preparation engine is also equipped with features such as bookmarking, tagging and rating the questions.

Having taken the tests in the classical way, by using Microsoft courses, MSDN and books for prep, I find uCertify’s product quite relevant and helpful. Looking back if I’d have to prepare for the future exams, I’d use this tool in association with recommended text to help gauge the learning.

For details and trial downloads, please visit UCertify's website.

• Aspect Oriented Programming in .NET, an Introduction with ASPECT.NET
  
  Aspect Oriented Programming (AOP) deals with factorization in code i.e. separation of common concerns, specifically cross-cutting concerns, as an advance in modularization. AOSD has been a popular trend in development for quite some time in other programming environments and IDE’s however it’s scope and exposure is limited among .NET developers. This session is focused on getting developers a deeper understanding of what AOP is all about and how to use it in their everyday development. Aspect.NET is the framework used for this presentation.
  
  
• Collaborative Filtering 101 – An Introduction with SQL Server 2008 BI
  
  "We have recommendations for you!". How do movies, social networking, books and e-commerce websites make recommendations? What algorithms and techniques are used behind the scenes?. In this session we will discuss collaborative filtering. Collaborative Filtering (CF) is defined as profiling or classification of information based on specific entity relationships i.e. making automatic predictions (filtering) about the interests of a user by collecting likelihood information from many users (collaborating). The underlying assumption of CF approach is that those who agreed in the past tend to agree again in the future. For example, a collaborative filtering or recommendation system for music tastes could make predictions about which music a user should like given a partial list of that user's tastes (likes or dislikes).
  
  
• Using ASP.NET MVC to build a blogging engine in 60 minutes or less.
  
  MVC is a framework methodology that divides an application's implementation into three component roles: models, views, and controllers. ASP.NET now has built-in support for MVC style development and this session is an introduction to using this technique for building a sample application, a blogging engine. This session will elaborate on differences between traditional ASP.NET post-back style development versus the routes and REST architecture based thinking around MVC.
  

This is a TIP, Twitter Initiated Post. Yes, the following message from Scott Hanselman was the tipping point for me to write this.

“ 4 of the 10 largest sites on the internet are ASP.NET. Live.com (#3) MySpace.com (#5) MSN.com (#7) Orkut (#10)”

This weekend at a social event, friend of a friend (fof) brought up an interesting topic; their company which shall remain nameless is moving away from ASP.NET / SQL / Win 2K3 to JSP / JBOSS / MySQL / Linux platform because apparently someone have told them that ASP.NET does not scale. Also, they have been having slowness issues with their website during the time of high traffic. They are in movie business so slowness during high traffic means loss of revenue when people cannot buy tickets and have to go for alternatives. But of course it has nothing to do with the underlying technology so I had to ask the obvious, "Was it designed to be scalable?"

A prototype which works on two machines may not be the perfect solution for large scale websites when no single point of contact can be a bottle neck, you have to make sure that the connections are handled properly and threads are available when needed.

So I had to defend the ASP.NET honor, counter that false belief therefore I tried explaining to the fof that this view about ASP.NET being unsuitable for enterprise systems is completely untrue. ASP.NET scales just fine, and that’s why four out of ten largest sites are in ASP.NET despite the fact that other platforms have been around longer and competing with free is a lot harder even with all the ROI results you can get. So it might be the configuration, connection pooling, number of open thread or a multitude of different issues which might be causing the problems they are having and can easily be resolved by an independent review.  Also, these issues can happen with any platform. I hope I am not starting a religious war here between .NET and Java web technologies but the truth is, they are both equally fine. Being a developer focused on Microsoft technologies, I have my personal bias towards ASP.NET. I have seen it working perfectly and there is enough empirical evidence to back up this claim. Some of the good ASP.NET optimization tips can be found here.

And here is an excellent Morgan Stanley guide on Internet Trends describing this and other zeitgeists; must see.

| View | Upload your own

• Yahoo Machine Learning
  http://research.yahoo.com/Machine_Learning
• Computational Advertising
  http://research.yahoo.com/Computational_Advertising
• Search Technologies
  http://research.yahoo.com/Search_Technologies
• Log file parsing Engine - Pig is an Apache incubator open-source project: http://incubator.apache.org/pig/
  http://research.yahoo.com/node/90

Exploring WCF 3.5 Tools - WcfSvcHost and WcfTestClient
Disucssing use of WCFSvcHost and WcfTestClient for Service hosting and testing.

Publishing RSS and ATOM Feeds using WCF 3.5 Syndication Libraries
This article focuses on using the WCF 3.5 libraries namely System.ServiceModel.Syndication namespace to create and publish an RSS and Atom feed from the same code base.

Enjoy!

Eric Simons, our director of QA and Offshore Development is visiting India right now and he is blogging about his experiences while there.

Interesting stuff! You can read it over here or subscribe to the RSS* Atom here.

I’m doing the following two sessions in the upcoming Fullerton Code Camp on Jan 25 and 26th.

.NET 3.5 Enhancements for WCF :: Syndication and REST Support

ASP.NET MVC Framework - An Introduction

Code Camp is Free! Feel free to come and join the geekiness.

Will it fly? How to Evaluate a New Product Idea

International Conference on Artificial Intelligence and Pattern Recognition (AIPR-08)

International Conference on Enterprise Information Systems and Web Technologies (EISWT-08)

International Conference on High Performance Computing, Networking and Communication Systems (HPCNCS-08)

International Conference on Software Engineering Theory and Practice (SETP-08)

International Conference on Theoretical and Mathematical Foundations of Computer Science (TMFCS-08)

Promote Research

Server in the Sky

First ACM International Conference on Web Search and Data Mining WSDM 2008, February 11-12, 2008, Stanford University

SVD Recommendation System in Ruby

MIT OCW Data Mining Course

How to Design Programs – MIT Press

How to Be a Good Graduate Student

Idle Scanning et al

Security and Hacking Documentation for Vulnerability Analysis

Dear Amazon.com Customer,

We've noticed that customers who have purchased or rated Expert Service-Oriented Architecture in C# 2005, Second Edition by Jeffrey Hasan have also purchased Workbook to accompany Anatomy & Physiology Revealed Version 2 CD by Robert Broyles. For this reason, you might like to know that Workbook to accompany Anatomy & Physiology Revealed Version 2 CD will be released on January 11, 2008.  You can pre-order yours by following the link below.

• How can I make my WCF WSDL same as Asmx WSDL? It looks different even when I use basic http binding with nothing fancy (security, reliable messaging..no WS* of course).
  
• I’m doing interop with Java and my stub generator is unable to understand the WSDL generated by WCF but it works just fine with Asmx WSDL, what am I doing wrong?
• What binding should I use which works best for interop?
• Dude, where are my types? 

etc...

The WSDL generated via WCF is different from traditional ASMX WSDL because they have split the schema into multiple segments (WSDL0, WSDL1....). This is the reason why when you look at the WSDL, you'll notice that the types seem to be missing! The stub building tools which come with other servers (Weblogic 6.0 for example) do not yet know this and do not iterate through the links specified in the schema by default, therefore you would not be able to make proxies out of it.

One of the big benefits of using WCF meta-data generation engine is that you can get XSD’s out of it too. Other platforms (read Java) tools understand and prefers XSD's (even though WSDL is the standard, XSD’s are cleaner IMHO). You can now easily generate them via a WCF service like follows.

• service.svc?xsd=xsd0
• service.svc ?xsd=xsd1
• service.svc?xsd=xsd2

Each of these xsd's has separated out contract, type and type definition.

Similarly you'd have WSDL0, WSDL1 and WSDL2 so

• service.svc?wsdl=wsdl0
• service.svc?wsdl=wsdl1
• service.svc?wsdl=wsdl2

Again, the wsdl's are separated to keep the contract, type and type definition (what is a double in interop environment) apart and not in giant one big file, which seems to be much easier but in essence its not.
Improving WCF Interoperability: Flattening your WSDL is an excellent article by Christian Weyer on increasing the interop bar. It explains the reasoning behind why a simple basic http binding service based WCF generated WSDL cannot communicate with its Java counterparts anymore and how to fix this problem.

In the next part I’ll discuss and share some examples and code samples.

• Will you cheat for me please, my dear computer: Text-mining and understanding system that can use the web to pass standard tests, e.g. SAT in World Literature-based discovery of drug X side effects History.
• Nip in the bud: Fraud detection based on company financial statements. (Can we find another Enron before it collapses?)
• Autonomous Tagging: Automatic tagging and classification of 1 billion digital photos on the web.
• Social Networking 2.0: Mining user behaviors in interactions with multimedia data and use the knowledge extracted in this process to anticipate future behaviors or to diagnose medical or psychological conditions of the users. This generally falls under the area of Crossing the semantic gap between multi-media data and semantics
• Where do I belong?: Link mining Challenge (extracting graphs describing entities and relationships from unstructured data)
• Lots of Traffic!: Estimating large dataset predictive model - from 833 traffic sensors in the Chicago metropolitan region and the goal is identifying anomalous traffic patterns.
• Gold in the Text: Entity extraction and autonomous text analysis from large scale unstructured text repository.
• And of course the genetics side, mining the proteome (Large-scale databases analysis from sequencing projects, micro array studies, gene-function studies, protein-protein interactions, comparative genomics, structural biology, and open source journal articles)

Also, the other areas of research interest mentioned in the data mining literature are  

• Parallelization of data mining algorithms.
• Designing and developing scalable algorithms to operate on massive data sets.
• distributed data mining; multiple topologies (local data, distributed app and so on …)
• Standardizing the languages, underlying protocols, and application level integration for data mining and predictive modeling.
• Systems to promote preserving privacy and security in the data mining.
• Visualization of large datasets; mapping their corresponding associations, hierarchies and underlying patterns.
  

Visual Studio 2008 VHD makes life a lot easier if you want to explore the upcfeature-set of  C# 3.0 compiler. Make sure to get the differencing image Visual Studio Code Name Orcas Base Image else you’d be getting the following error.

"OrcasBeta2_VSTS" could not be started because a disk-related error occurred.

Feeling bad for fellow .netters (Yes Ben, its you) who had to clean install the beta2 upgrade from beta1. For me, it was plugging in the new machine :) VM Rocks..errr...I meant virtual PC VM, not the other VM.

Anyhow, support for lambda expressions is one of the gems in C# 3.0, as defined in the C# 3.0 Language Specification

lambda-expression:
(   lambda-parameter-list_opt   )   =>   lambda-expression-body
implicitly-typed-lambda-parameter   =>   lambda-expression-body

This is essentially the same thing you’d do in CLisp, the function definition, application and recursion, all in quite elegant way. Therefore, the implication inherently defines the method without any explicit declaration. So you can say

x => x + 1 which translates to a function that takes one argument, c, and returns the value x + 1.

A less trivial example is as follows.

List<string> XFilesEpisodes = new List<string>();
XFilesEpisodes.Add("Little Green Men");
XFilesEpisodes.Add("The Host");
XFilesEpisodes.Add("Blood");
XFilesEpisodes.Add("Sleepless");
XFilesEpisodes.Add("Duane Barry");
string episodeMatch = XFilesEpisodes.Find(p => p.Equals("Sleepless"));
Console.WriteLine(episodeMatch);
Console.Read();

 string episodeMatch = XFilesEpisodes.Find(delegate(string name) {
                            return XFilesEpisodes.Equals("Sleepless");
                        });

The lambda methods evaulated within closures; these anonymous delegated blocks are not called closures because their understanding would give closure to your intellectual feat but because they can access the local members. As defined on precious wikipedia (since I can never find my PL book handy), a closure is a function that is evaluated in an environment containing one or more bound variables. When called, the function can access these variables.

Now on the IL level, this can be seen as a static predicate class of anonymous delegate (hence the expansion)

.field private static class [mscorlib]System.Predicate`1<string> '<>9__CachedAnonymousMethodDelegate1'

.custom instance void [mscorlib]System.Runtime.CompilerServices.CompilerGeneratedAttribute::.ctor() = ( 01 00 00 00 )

It is defined as anonymous delegate in the general metadata and further performs a string comparison with virtual call (in bold below). 

.method private hidebysig static bool  '< Main>b__0'(string p) cil managed
{
  .custom instance void [mscorlib]System.Runtime.CompilerServices.CompilerGeneratedAttribute::.ctor() = ( 01 00 00 00 )
  // Code size       12 (0xc)
  .maxstack  8
  IL_0000:  ldarg.0
  IL_0001:  ldstr      "Sleepless"
  IL_0006:  callvirt   instance bool [mscorlib]System.String::Equals(string)
  IL_000b:  ret
} // end of method Program::'< Main>b__0'

The Main method is as follows which shows the generic list of items and then the actual call to CachedAnonymousMethodDelegate.
 
.method private hidebysig static void  Main (string[] args) cil managed
{
  .entrypoint
  // Code size       110 (0x6e)
  .maxstack  4
  .locals init ([0] class [mscorlib]System.Collections.Generic.List`1<string> XFilesEpisodes,
           [1] string episodeMatch)
  IL_0000:  newobj     instance void class [mscorlib]System.Collections.Generic.List`1<string>::.ctor()

.......

  IL_003e:  ldsfld     class [mscorlib]System.Predicate`1<string> VS.NET2008Features.Program::'<>9__CachedAnonymousMethodDelegate1'
  IL_0043:  brtrue.s   IL_0056
  IL_0045:  ldnull
  IL_0046:  ldftn      bool VS.NET2008Features.Program::'<Main>b__0'(string)
  IL_004c:  newobj     instance void class [mscorlib]System.Predicate`1<string>::.ctor(object,

                                                                                       native int)
 IL_0051:  stsfld     class [mscorlib]System.Predicate`1<string> VS.NET2008Features.Program::'<>9__CachedAnonymousMethodDelegate1'

  IL_0056:  ldsfld     class [mscorlib]System.Predicate`1<string> VS.NET2008Features.Program::'<>9__CachedAnonymousMethodDelegate1'

  IL_005b:  callvirt   instance !0 class [mscorlib]System.Collections.Generic.List`1<string>::Find(class [mscorlib]System.Predicate`1<!0>)

  IL_0060:  stloc.1

  IL_0061:  ldloc.1

  IL_0062:  call       void [mscorlib]System.Console::WriteLine(string)

  IL_0067:  call       int32 [mscorlib]System.Console::Read()

  IL_006c:  pop

  IL_006d:  ret

} // end of method Program::Main

The extension methods are a very powerful feature. Think of them as highly sophisticated version of poor man’s global overrides. For instance, in the example below I’d convert all the temperature by “extending” the method on the double type which does not define a Convert method.

namespace VS.NET2008Features
{
   static class Program
   {
       static void Main(string[] args)
       {
           double temperature = 0;
           temperature.Convert();
           Console.Read();
       }

       static void Convert(this double temp)
       {
           double tempF = (9/5)* temp +32; // 9/5.0 not done on purpose.
           Console.WriteLine(tempF);
       }
   }

}

For a simple program like this, the underlying IL and reflector code looks quite interesting. First of all, note the clever IL as it will evaluate the 9/5 as 1 (integer value) since it is not a double calculation (9/5.0 would have been). Also, the extension attribute is added and redirection to this new extended method is done at the IL level. Clever eh?
 

private static void Convert(this double temp)
{
    double num = (1 * temp) + 32;
    Console.WriteLine(num);
}

The extension method is depicted here by setting the extension attribute in the CLR. The call to the Convert gets translated and redirected to VS.NET2008Features.Program::Convert(float64)

Scott Guthrie defines it best, from both developer and framework architect prospective. And till then, I'm waiting for Jeffrey Richter's addendum to CLR via C# which would discuss the C# 3.0 enhancements.

The classes are scheduled every Thursday, 6:35 p.m. - 10:00 p.m. from 8/16/2007 - 9/20/2007 (6 mtgs.). Further details and the enrollment details can be found at the following link.

WCF Course -UCSD Extension
CSE-40114  Credit: 3 units
“.NET 3.0 introduces Windows Communication Foundation (WCF) providing a service-oriented programming model for distributed application development. This course will use the C# programming language and cover designing, implementing, configuring and hosting WCF servers and clients to leveraging this new communication stack and its protocol facilities for security, reliability, transactions and other services.”

The course’s text book is Michele Bustamante’s “Learning WCF – A Hands on Guide”. The source code for the book labs can be downloaded from here. We will be discussing a lot of interesting topics in WCF and connected systems so spread the word.

Document Literal vs. RPC Encoding is the big endian-little endian of connected systems. There has been a lot said about the differences (see reference section for details) however this post is about how elegant it looks in WCF. Traditionally we had two ways to represent them;

As a web method

Or as a SoapDocumentMethod

With WCF's seperation of Data and Service Contracts, it has become much more cleaner and clear.

and the one way attribute as an operation contract.

public interface IOrderEntry
{
[OperationContract(IsOneWay = true)]
void PlaceOrder(PurchaseOrder order);
}

Remember that RPC/encoded is not WS-I compliant!

Following is difference in the output from the two different

References

• RPC/Literal and Freedom of Choice (MSDN)
• Web Services Encoding and More (MSDN)
• Which style of WSDL should I use?
• Patterns and Strategies for Building Document-Based Web Services
• wsdl:binding "style", "use", and message format
• Understanding WSDL
• Doc/Literal/Bare
  

• Powershell Live 
• PowerScripting Podcast
  

As I was driving down to work this morning trying to catch up on .NET Rocks, I started listening to ASP.NET Scalability Panel Tech-Ed 2007 show. This is an excellent panel discussion; highly recommended to anyone who writes an ASP.NET web application. The panelists include Stephen Forte, Kent Alstad and Rob Howard who talked about caching, SQL Profiler, non reproducible timeouts, web server optimization and the business case for performance etc; you surely don’t want to miss it.

Also during the SoCal Code Camp I attended a talk by Richard Campbell on ASP.NET scalability (also dubbed as “Killing web servers for fun and profit”) and there was a equation he mentioned during the presentation which is also available as part of Stephen Forte’s PDC slides. The equation goes as

R ≈ (Payload / Bandwidth ) + AppTurns(RTT) + Cs + Cc

Where

AppTurns is the turn count required to generate a user response

Bandwidth is the minimal bandwidth (bits per second) across all the network links between the user and the application server.

Cc (Compute Client) is the total processing time (seconds) required by the client device.

Cs (Compute Server) is the total processing time (seconds) required by the server(s).

Payload is information content (bytes) that must be delivered to/from the user’s device.

R is the response time, which is the elapsed time (seconds) between a user action and the system response (client, network, server),

RTT is the round-trip-time (seconds) between the user and the application server.

Aside from figuring out the weakest link in your system using the above equation, you should definitely check out the following resources if interesting in making your website more “responsive” and effective. Feel free to drop me a line (adnanmasood at gmail dot com) if you have any interesting problems in this area and I’d love to assist.

References

• Web Page Analyzer
• Speed Test
• .NET Rocks Show - ASP.NET Scalability Panel
• Red gate ANTS tool
• Strangeloop Networks
• Scaleout Software
• Akamai: Web Application Acceleration and Performance
• Improving .NET Application Performance and Scalability (Patterns & Practices)

 
I gave a talk on Web Service software factory in the SoCal Rock and Roll code camp  this afternoon. It was well received (at least I am under this impression since there was nothing thrown at me, ok yes, one tomato, big deal!) and following are the links to the WCF resources discussed.

• Web Service Software Factory Known Issues
• Service Factory Community
• Web Service Software Factory – Patterns and Practices
• Web Service Software Factory – Videos (Highly Recommended)
• Service Factory Hands on Lab (Both for ASMX and WCF)
• Web Service Security Guide

Presenting at the SoCal rock and Roll Code Camp

I don’t know about you folks but my memory is quite lousy. At the ripe age of 26, it’s not as good as its use to be and hence I usually end up looking things up like when trying to use Log4NET. My favorite destination is either the Apache website or Nauman’s article on Log4NET. So here is a simple five step process of adding logging to an application. For times when MS logging app block seems like overkill and console writing to a text file is not an option, Log4NET is the way to go.

So without further ado

Step 1. Add the following at the top of the application config file. This tells the config engine that there is a Log4NET section

<?xml version="1.0" encoding="utf-8" ?>
<configuration>
  <configSections>
    <section name="log4net" type="System.Configuration.IgnoreSectionHandler"/>
  </configSections>

 
Step 2. Add the following before/after appsettings.

<log4net>

    <appender name="MyApplicationLogFileAppender" type="log4net.Appender.RollingFileAppender">
      <param name="File" value="log\\MyLogFile"/>
      <param name="DatePattern" value=".yyyy-MM-dd-tt&quot;.log&quot;"/>
      <param name="AppendToFile" value="true"/>
      <param name="RollingStyle" value="Date"/>
      <param name="StaticLogFileName" value="false"/>
      <layout type="log4net.Layout.PatternLayout">
        <param name="ConversionPattern" value="%r %d [%t] %-5p %c - %m%n"/>
      </layout>
    </appender>
    <root>
      <level value="DEBUG"/>
      <appender-ref ref="MyApplicationLogFileAppender"/>
    </root>
  </log4net>

Step 3. Add the following in the AssemblyInfo.cs

[assembly: log4net.Config.DOMConfigurator(Watch = true)]

 Step 4. Add the following at the top of your class

public static ILog log = LogManager.GetLogger(System.Reflection.MethodBase.GetCurrentMethod().DeclaringType);

Step 5. Add the following line in the code where you want logging to be done.

log.Info("Program Started " + DateTime.Now.ToString ());

This will log the info in the MyLogFile in the log folder.

The above steps provide a cookie cutter recipe for jump start using Log4NET. For further information about what the above config sections actually mean, please see the following reference articles below.

References

Apache log4net: Home

ONDotnet.com -- Using log4net

Jeff Bergman, a friend and co-worker started his blog titled “Thoughts on the science of Computing”. He is currently pursuing his Masters from UCLA in Computer Science and has some great thoughts on topics like algorithms, search, data mining and computational heuristics.

His blog can be found at

http://jeffbergman.com/cs/blogs/csjeff/default.aspx

and the RSS can be added from here.
http://jeffbergman.com/cs/blogs/csjeff/rss.aspx

…

I've always lived that way. It's nice, it's pleasant if you can do it. I'm lucky in my life that I can do this.”

-Richard Feynman
Los Alamos From Below: Reminiscences 1943-1945 - Caltech

Feynman said this after he pretty much told, physics legend Neil Bohr, off. During a discussion, Feynman said following about Bohr’s idea

"No, it's not going to work. It's not efficient. . . Blah, blah, blah."

and

"That sounds a little bit better, but it's got this damn fool idea in it."

In the above mentioned Feynman's words, there is a good lesson for everyone; especially for those of us who call themselves Software Engineers. When you are providing input to someone about architecture, design or anything CS in general, be very honest. If someone tells you how to do “software things” and you think the specified method is flawed, speak up. Always be worried about computer science and software engineering like Feynman was about Physics. Just because someone has 15-20 years of experience, doesn't mean that every idea they'd propose is going to be great. Don't argue for the sake of argument but if you see a problem, point it out. There are more than enough people in the world who would think that having a flash button is far better than a static gif because "you can feed text to it via a parameter and it would become whatever button you want it to be". Might sound cool but generally speaking, this is definitely not a good enterprise design idea (instantiation cost, compatibility etc). If a senior developer wants to put config settings in the database, and you see that they don’t belong there, question it. If that person also loves technology, simplicity and good design, they would concur with you once they see their mistake. Otherwise, you’ll always have the option of singing “I told you so” when the thing starts crumbling. Without courting with your ego, fight till you can and make your case from a technology standpoint. Do not sign off on a technical design just because the next person did. It’s important that you practice your judgment in software design; such people are definitely an asset to every software team.

• Becoming an Experienced, Invaluable Web Analyst
• The 2007 International Conference on Data Mining
• Doing NTLM with Firefox
• FireBug, a FF Javscript debugger
• Famous Equations and Inequalities
• Things I wish I'd known before I installed VS 2005 Service Pack 1
• ASP.NET 2.0 Tips, Tricks, Recipes and Gotchas
• “ASP.NET: How Do I?” videos
• Rethinking your business validation rules
• Techno-Science
• Microsoft adds behavioral targeting
• AI and Computing Quotes
• Coding4Fun Holiday Gift Guide

We definitely need something better than the following pseudo code

String.Replace (String.IndexOf(<companyName>), <companyname>.StockSymbol) 

A text based context sensitive substitution which actually recognizes when word “Amazon” as in the jungle and Apple as in the fruit.

In the January 2007 MSDN magazine, James Waletzky discussed the estimation from a SWAG (Silly Wild Ass Guess) prospect and then talked about Wideband Delphi for estimation. Rapid estimations are a reality of today’s fast paced development environments especially in the vertical markets where software development is means for a tool and not the core business of the organization, What would be the most appropriate method to estimate the time frame which is more efficient than guessing M&M’s in a jar at a Rubio’s and less time consuming than a function point estimation via Matson, Barnett, and Mellichamp Model for E = 585.7 + 15.12 FP?. Let’s see what popular methodologies have to offer.

As a principle, all software estimation methods share the Gödel's incompleteness theorems as their fundamental axiom i.e. the modeling of system is essentially not perfect. Traditional software engineering approaches to estimation include decomposition techniques like LOC (lines of code) based estimations, function point based estimates, process-based estimates, use case oriented estimates etc. These techniques essentially quantify the task breakdown via different methodologies and then estimate the time based on how long a single entity would take. Sounds like common sense, right? The empirical estimation models include function oriented metrics, COCOMO model and its variations. Aside from these heavy weight processes, Agile methodology describes the estimation process as follows.

• Each user scenario is considered separately for estimation purposes
• The scenario is decomposed into the set of functions and engineering tasks required to develop them.
• Each task is estimated separately, based on historical data, an empirical model, or “experience”
• Estimates for each task are summed to obtain an estimate for the scenario.
• The effort estimates for all scenarios are summed for a given increment to obtain the increment estimate

This sounds more realistic, five step process and you are done with it. As we already know as Moore et al described “Our preliminary results suggest that complicated methods may not necessarily yield a more accurate estimate, particularly when developers can incorporate their own intuition into the estimate”, or simply that making fancy estimations won’t make you more accurate.

In order to determine the rough order of magnitude (ROM), i.e. a rough estimate of the number of person-days to complete the task, I prefer the agile way. A typical workflow would be as follows.

Identify: Go through the specs (purpose statement/business requirement document or any other document) you have which identifies what needs to be done. Identify how you would do it in the scope and boundaries of current system.

Divide and Factor: With a rough sketch of identified tasks, factor out the similarities and what components can be shared across the board. Split the tasks into smaller pieces until it reaches ‘sane atomicity’. Try to think in service context so the responsibility is centralized instead of shared across the components, it helps the bottom line.

Raise the Unknown: Always identify unknown as risk, it helps. If you don’t know that the new mail server can handle bounce back processing for your application or you have not evaluated the ISO format file parser library performance yet, please make note of it.

Estimate:  Take an educated guess of how long it would take you to complete the tasks. If you are working as a team, knowing their respective skillets, ask fellow developers for their estimates. Most likely they would be the ones working on some of the components too. Find out the median, pad it to include the process factors (documentation, release notes, collaboration delays) and here your ROM guesstimate.

Repeat: As you identify the tasks more and more clearly, re-iterate through the steps for accuracy and effectiveness.

This process does not take too long, depending on the size of project it could be less than an hour and its much better than “hmmm, there are 1000 M&M’s in this bottle”. This is more like “In one square inch of this jar, I can see almost 20 chocolate goodies. Providing the curve and integrating below it, I think it would have…”, you got the idea. You can re-calibrate it with future iterations as it would be a good learning exercise in estimation. This method will get more and more concrete proportional to your specifications as they say, requirements are like water. They're easier to build estimate when they're frozen.

“I love deadlines. I like the whooshing sound they make as they fly by.”
-- Douglas Adams

References and Further Readings

• MIT Blog on Software Cost Modelling
• Agile Software Development for the Entire Project
  Granville Miller, Microsoft
• The Classic Story of the Pig and Chicken
• Bayesian Analysis of Empirical Software Engineering Cost Models
• An Empirical Study Using Task Assignment Patterns to Improve the Accuracy of Software Effort Estimation
• Modern Empirical Cost and Schedule Estimation Tools
• Empirically-Guided Software Effort Guesstimation
  Philip M. Johnson Carleton A. Moore
  Joseph A. Dane Robert S. Brewer
  Collaborative Software Development Laboratory
  Information & Computer Sciences Department
  University of Hawaii

Joe White's Blog states Randy Miller's MSF for agile salient points some of which are inspired from context driven school. Interesting ideas and quite true in their regards

1. The value of any practice depends on its context.

2. There are good practices in context, but there are no best practices.

3. People, working together, are the most important part of any project's context.
   

4. Projects unfold over time in ways that are often not predictable.

5. The product is a solution. If the problem isn't solved, the product doesn't work.

6. Good software testing is a challenging intellectual process.

7. Only through judgment and skill, exercised cooperatively throughout the entire project, are we able to do the right things at the right times to effectively test our products.

Reference: Seven Basic Principles of Context Driven School

With session titles like “Put Testing Where It Belongs--At the End”, “The Joy of Silence: Cube Farm Designs That Cut Out Conversation” and “Unfactoring from Patterns: Job Security through Unreadability”, this is a must attend conference for software developers and architects, not.

The registration page is riot, as it states: 

“We're sorry but registration is not yet ready. Our software developers have a really wonderful design. They're almost done entering it into it a UML tool. They've told us not to worry and that finishing it will be "trivial" because "all that's left is the coding."

Credit goes to Richard Hundhausen for the mention, go agile!

MSR Turned 15
http://research.microsoft.com/aboutmsr/15years/default.aspx

Grid Computing: Powers Unfiltered by John Powers
http://powersunfiltered.com/

Grid Computing:  dan ciruli's West Coast Grid
http://westcoastgrid.blogspot.com/

Grid Computing: A Day in the Life
http://krgreenlee.blogspot.com/

And who would not love LISP, it’s so simple and self explanatory

by Jeff Bergman

 
; function select_min(L:list of integers):integer
; returns the minimun interger in the  list
;

; Invariants:
(defun select_min(L)
  (cond ((null (rest L)) (first L ))
            ( t (let ((min (select_min (rest L))))
                (cond ((<= (first L) min) (first L))
                          (t min))))))

Andrew Binstock, principal analyst at Pacific Data Networks recently wrote in SD Times about free online book resources. Following are my picks from his list.

• Free online computer science and engineering books and lecture notes
• An extensive list of free math books on the Net.
• Programming language theory texts online
• Directory of Open Access Journals
• Networked Computer Science Technical Reference Library

I’ve received the following email via David Riggs, the editor of Asp.net pro about my recent article Naive Bayesian Classification Using C# and ASP.NET. This is from a careful reader who spotted a typo in the article.

The same typo was spotted by Dr. Homayoun Seraji as well.

Thank you both.

Dear ASP.Net Pro editors,

I believe there may be a small typo in one of your formulas in your September 2006 article Naïve Bayesian Classification Using C# and ASP.NET.  If you look at the bottom of the left column on page eight, you'll see the formula:

       P(A|X) = (0.3) (0.5) / 0.037 = 0.405405... ~40.5%

The 0.3 should be 0.03 and the formula should read:

       P(A|X) = (0.03) (0.5) / 0.037 = 0.405405... ~40.5%

Normally I'm not such a typo-Nazi - great article, by the way.  I may have to start subscribing if you're going to do more math and statistically/probability-related articles.

Greg Pyatt
Information Systems Program Analyst
The Boeing Company

·         Stored Procedure Object Interface Layer (MSDN)

·         Dan ciruli's West Coast Grid

·         Microsoft Robotics Studio (Channel 9)

·         Robotics Studio August 2006 CTP

·         Robotics Home Page: http://msdn.microsoft.com/robotics

·         Robotics Group team Blog: http://blogs.msdn.com/MSRoboticsStudio/

·         XNA Game Studio Express (Beta) Download

·         MyGeneration (ORM Tool)

·         The Most Intelligent Java IDE- IntelliJIdea (Highly Recommended  for those who didn't really liked Eclipse)

·         Artificial Ignorance - Anand Iyer's Blog (SoCal Developer Community Champion)

·         A Really Simple Service Bus (An effort in simple ESB)

·         Serializer Robot Controller

·         Watir: Web Application Testing in Ruby

·         Scott Hanselman's Computer Zen - Integrating Ruby and Watir with NUnit

·         Boomerang decompiler

·         Windows® XP Urdu Language Pack اردو مواجہ پیک

·         A good collection of algorithms in C# and Java

·         Internet security zone blog

·         Open Source 3D engine in .NET

Beside Web 2.0 and AJAX, Service Oriented Architecture is most probably among the top ten tech buzz words of this year and can also be regarded as the term with most diverse array of definitions or the most misunderstood concept. Currently there are several books available in the market with word SOA in their titles however if you are not looking for a 10,000 ft overview of service orientation but want a practical guide which walks you through step by step in the journey of loosely coupled architecture & design, Jeffery Hasan and Mauricio Duran’s “Expert Service-Oriented Architecture in C# 2005” is the perfect book for you

.

In this book, SOA is defined as “architecture based on loosely coupled component that exchange messages” but it doesn’t stop here. Jeff explains in depth what each and every term in this definition mean and how to practically implement it using WSE 3.0. Service oriented architecture is a concept and in order to implement its features like interoperability, declarative policies, contracts, message security, identity, trust and transactability, one would need a concrete framework such as WSE. Readers who have found this book more on WSE and less on SOA are usually looking at SOA from a much abstract prospect while the book is very much hands on emphasizing more on practical implementations. Author has tried to convey the ground realities by transferring the theory into practice.

This 250 page book is divided into nine chapters and in mainly three logical sections. 

• Introduction to Distributed Computing, Messaging and SOA (Chapter 1-4)
• WSE 3.0 and SOAP   (Chapter 5-8)
• WCF and future directions (Chapter 9)

Clarity is the one of the major traits of Jeff’s writing; the book is written in no-nonsense developer friendly language; it employs effective use of diagrams, selected listings and notes to convey its message in short and concise manner. The book also addresses common SOA myths and tries to bust them. For instance, how can MSMQ and web services co-exist and can even be used in the same enterprise solution? For most people, these technologies including Enterprise Services and remoting sound like rivals. Author debunks these ideas by providing examples and scenarios which explain service orientation as an architectural concept instead of a development framework. This book goes beyond the common realm of design and further explains that SOA is not tied together with SOAP/XML web services but can be implemented using any distributed communication protocol providing that it supports loosely coupled message exchange and other primary SOA traits discussed above.

Web Services Enhancements for Microsoft .NET (WSE) is a library provided to extend the Microsoft .NET Framework capabilities of handling web services and to keep pace with the evolving Web services protocol specifications. In “Expert Service-Oriented Architecture in C# 2005”, author does not discuss merely the turn key scenarios provided but provide in depth elaborate study of WSE 3.0. The book does not contain any enterprise level case study but some concrete and small examples to keep focus on concepts without turning it into a WSE cook book. Author makes the business case of WS-Specifications providing its reader a solid understanding of specifications like WS-Security, WS-Secure Conversation, WS-Addressing, WS-Reliable Messaging and WS-Policy. Author notes that transactions are not implemented in WSE, making it a live specification and emphasizing on the need of windows communication foundation (formerly code name Indigo).

Author tries to make sure that the readers are not merely using web services as another RPC mechanism but are designing good robust architectures. Therefore in the design patterns of building SOA, he urges on writing type assemblies (type and methods), business assemblies, web services and client (test harness) separately and follows the same design pattern through out the book. Having the dedicated typed assemblies is one of the best practices I’ve found through experience in distributed application development as it eases the complex type communication and marshalling lot easier.

Since I’ve read the first edition and have worked with WSE 3.0 in the past, I jumped right to the chapter 9 regarding WCF and was not disappointed. Jeff has done a fairly good job explaining the nuts and bolts of Windows communication foundation however it’s still the tip of iceberg since the book is not mainly about windows communication foundation. (For a detailed study of WCF, please check David Pallman’s Programming WCF (Indigo) by MS Press.)

If you are development manager who wants to grasp the WSE concepts to know what can be done in the world of advance web services, chapter 1-5 will give you well rounded foundation and understanding while chapter 9 would help you building a migration strategy to WCF.  For an application architect or an API developer, you’ll find this book specially interesting and valuable because of its architectural approach. As an example, there are numerous times that as a developer you build request and response objects for your web service components with end to end security in mind. Client informs you that SSL won’t be a silver bullet anymore since they have to do re-routing between different layers of application process. Sound like user name token manager authentication and SOAP headers to me, no problem! Open page 161 and code it away. It will enlighten you about the differences between HTTPS and WS-Secure Conversation, very well done. Even your IT folks may admire a complete step by step direction of setting up Kerberos authentication on Windows Server 2003.

The source code provided with the book is converted using the Visual Studio.NET 2005 wizards, contains UpgradeLog.XML and not really reflect the WSE 3.0 changes. For instance building Chapter8\WSSecureConvService\App_Code\StockTrader.asmx.cs throws warnings such as

Microsoft.Web.Services3.SoapContext.Security' is obsolete: 'SoapContext.Security is obsolete. Consider deriving from SendSecurityFilter or ReceiveSecurityFilter

and

// Use the SCT to sign and encrypt the response
SoapContext responseContext = ResponseSoapContext.Current;
responseContext.Security.Tokens.Add(sct);

throws exception as 'Microsoft.Web.Services3.SoapEnvelope' does not contain a definition for 'IsSoapEnvelope'

As it’s not in the WSE 3.0 specs (See MSDN reference).

For a developer and architect alike, I find this to be of great use and I would suggest it to all of you interested in service oriented architecture or distributed application development using WSE 3.0 to give this SOA page turner a try. Notwithstanding, those willing to invest the time will be richly rewarded with the most intelligently written SOA book I can recall reading in a while.

Highly recommended!

The source code can be downloaded from the following link
Download Source Code File
Also, you can read a sample chapter from here.
Ch. 04 - Design Patterns for Building Service-Oriented Web Services

Beside the slick look and tabbed browsing, two of the best features I like about IE 7 is its RSS feeds handling and website thumbnails.

The clean and well organized RSS aggregator is much better from usability prospect than anything I've tried before. Import your OPML's and try it out for yourself.

The website thumbnail view tells you all about what websites are open, a live version of print preview.

and yes, its ironic to have www.ie7.com pointing to FF and Clipboard exploit (Retrieve Clipboard Text To A Web Page With Javascript) still there.

References

• IEBlog
• Internet Explorer 7: Home
• Intelligent browsing with Internet Explorer 7
• Add-Ons for Internet Explorer
• Internet Explorer Administration Kit 7 Beta 2
• Readiness Toolkit for Developers, Testers & ITPros
• HTTPS Security Improvements in Internet Explorer 7
• Finding Security Compatibility Issues in Internet Explorer 7