Here's a C# example of how to do it using the managed api. Note that the SelectAll method accepts a single array.
Code:
// Get a selection set of ents whose color is not ByLayer.
TypedValue[] tvls = new TypedValue[2];
tvls.SetValue(new TypedValue((int)DxfCode.Operator, "!="), 0);
tvls.SetValue(new TypedValue((int)DxfCode.Color, 256), 1); //Color 0 = ByLayer
SelectionFilter sf = new SelectionFilter(tvls);
PromptSelectionResult PSR = Active.Editor.SelectAll(sf);
If you still want to use the COM method, the example in this Help topic filters for a circle. Note that the Select method takes variant arrays, so you have to assign your filter data to variant arrays first before trying to use it in the Filter argument.
BTW, to put your code in a CODE window, click on the Go Advanced button, then select the # button to insert CODE tags.